How do I?

User Tools

Site Tools

You are here: start » techstaff » aicluster-admin
techstaff:aicluster-admin

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
techstaff:aicluster-admin [2021/02/10 13:23] – kauffmantechstaff:aicluster-admin [2021/02/23 19:58] (current) – kauffman
Line 1: Line 1:
 ====== AI Cluster Policy Description ====== ====== AI Cluster Policy Description ======
 +===== TODO =====
 +  - There are multiple methods used to calculate priority reflected on the spreadsheet.
 +  - Double check the math for correctness.
 +  - Does the math reflect our intent? I think it does.
 +  - Multiple methods in calculating priority are reflected (blue to purple cells: ''%%har_priority%%'', ''%%normalized_priority%%'', ''%%weighted_normalized_priority%%''
 +  - Choose one calculation to use. This decision doesn't prevent us from changing it later if we find another calculation works better.
 +  - If you have a suggestion: Please show us the work in a new column or by cloning the sheet.
  
- +===== Contribution Tracking and Priority Calculation Tool =====
-After various iterations we (Bob, Har, and I) believe to have an +
-implementation of the policy that meets the requirements discussed +
-previously.+
  
 [[https://docs.google.com/spreadsheets/d/15o3jZOVqU84hMevIKnLKj8DhFYaQUdPdOYE1_itucqk|AI Cluster Tracking and Priority calculation spreadsheet]] [[https://docs.google.com/spreadsheets/d/15o3jZOVqU84hMevIKnLKj8DhFYaQUdPdOYE1_itucqk|AI Cluster Tracking and Priority calculation spreadsheet]]
Line 10: Line 14:
 AI Cluster committee members have access. AI Cluster committee members have access.
  
 +==== Sheet usage ====
 +  * Red: Do not edit
 +  * Green: user input (This will be Techstaff 95% of the time)
 +  * `groups` sheet:
 +    * contributions get assigned a POSIX group. Group must have a primary contact, who then gets to set members for that group.
 +    * calculates contribution amount for use in `contrib-priority`.
 +    * tracks group name and primary owner
 +  * `log` sheet
 +    * All contributions will get entered here.
 +    * Hardware contribution gets converted to USD by techstaff. A receipt of the purchase is good starting point.
 +  * The group 'cs' is calculated on the spreadsheet but doesn't actually get any priority set. 'general' is for all CS users. It needs to be part of the total sum but is treated as a special case with 0 priority.
 +  * `contrib-priority` calculation references contrib amounts calculated in `groups`.
  
-==== TODO ==== 
-  - There are multiple methods used to calculate priority reflected on the spreadsheet. 
-  - Double check the math for correctness. 
-  - Does the math reflect our intent? I think it does. 
-  - Multiple methods in calculating priority are reflected (blue to purple cells: har_priority, normalized_priority, weighted_normalized_priority 
-  - Choose one calculation to use. This decision doesn't prevent us from changing it later if we find another calculation works better. 
-  - If you have a suggestion: Please show us the work in a new column 
-or by cloning the sheet. 
  
  
  
-===== Details ===== +===== Understanding Slurm Fairshare and Priority/Multifactor ===== 
-By default the cluster uses a [[https://slurm.schedmd.com/fair_tree.html|fair share algorithm]] with [[https://slurm.schedmd.com/priority_multifactor.html|multiple factors]] to adjust job priorities.+Slurm comes with built in tools to calculation fair share priorities without anyone needing to do anything special. The cluster uses a [[https://slurm.schedmd.com/fair_tree.html|fair share algorithm]] with [[https://slurm.schedmd.com/priority_multifactor.html|multiple factors]] to adjust job priorities.
  
 +==== How Slurm calculates Job priority ====
 Generally Slurm will use this formula to determine a jobs priority. Generally Slurm will use this formula to determine a jobs priority.
 <code> <code>
Line 41: Line 50:
 </code> </code>
  
-The factors on the left that start with ''%%Priority*%%'' can and are set in the Slurm cluster configuration file (found on all nodes at /etc/slurm-llnl/slurm.conf):+The factors on the left that start with ''%%PriorityWeight*%%'' can and are set in the Slurm cluster configuration file (found on all nodes at /etc/slurm-llnl/slurm.conf). The factors on the right are calculated based previous job submissions for that particular user.
  
 <code> <code>
Line 55: Line 64:
 PriorityFavorSmall=YES PriorityFavorSmall=YES
 </code> </code>
-**Note that this example may not be up to date when you read this.+*Note that this example may not be up to date when you read this.
  
-We adjust those priorities with partitions for those who have donated either monetarily or with hardware. Hardware donations get converted a monetary value when logged on the spreadsheet.+===== How we modify job priority to favor contributors =====
  
-Every contribution gets assigned a POSIX group, hereon refered to as ''%%$group%%''.+  * We adjust those priorities with partitions for those who have donated either monetarily or with hardware. Hardware donations get converted a monetary value when logged on the spreadsheet. 
 +  * Every contribution gets assigned a POSIX group, hereon referred to as ''%%$group%%''.
  
  
- +Here is a version of the partition configuration as it stands now (2021-02-10).
-Here is a simplified version of the partition configuration as it stands now. +
 <code> <code>
 PartitionName=general Nodes=a[001-008] PartitionName=general Nodes=a[001-008]
-PartitionName=cdac-own Nodes=a[005-008] AllowGroups=cdac Priority=100+#PartitionName=cdac-own Nodes=a[005-008] AllowGroups=cdac Priority=100
 PartitionName=cdac-contrib Nodes=a[001-008] AllowGroups=cdac Priority=5 PartitionName=cdac-contrib Nodes=a[001-008] AllowGroups=cdac Priority=5
 </code> </code>
  
-^Partition^ Description^ +^Partition^Description^Priority^ 
-|general| For all users| +|general| For all users| 0 | 
-|${group}-own | Machines $group has donated | +|${group}-own | Machines $group has donated. Enabled when asked. | 100 | 
-|${group}-contrib | A method to give slightly higher job priority to groups who have donated but do not own machines.|+|${group}-contrib | A method to give slightly higher job priority to groups who have donated but do not own machines.| Variable based on spreadsheet calculation. |
  
 +The key thing to notice before you continue reading is that nodes can be added to multiple partitions. ''%%general%%'' and ''%%cdac-contrib%%'' can submit to all nodes but with different priorities.
  
-The key thing to notice before you continue reading is that nodes can be 
-added to multiple partitions. 
  
-'general' and 'cdac-contrib' can submit to all nodes but with different +==== Calculating -contrib partition usage ====
-priorities.+
  
-Priority is normalized in the sheet to be 0-100. +We do the following calculation to determine the ''%%*-contrib%%'' partitions usage over the past 30 days in comparison to total cluster usage.
- +
-Understanding the partition configuration: +
- +
-  - All users get access to partition ''%%general%%''. It has a default priority of 0. +
-  - Group 'cdac' has donated nodes ''%%a005-a008%%''. They always get top +
-priority on those machines (Priority=100). This means that at most they would wait 4 hours for their job to be +
-submitted. +
-  - 'cdac-contrib' or "${groupname}-contrib". Since 'cdac', in this example, has donated to the cluster they should get a higher priority on other machines as well. +
- +
- +
-We do the following calculation to determine the ''%%*-contrib%%'' partitions (cdac-contrib) usage over the past 30 days in comparison to total cluster usage.+
  
 <code> <code>
Line 104: Line 99:
 The percent will end up as an integer. The percent will end up as an integer.
  
 +There is a [[https://vcs.cs.uchicago.edu/kauffman/slurm-tools/blob/master/cluster_partition_usage.py|python script]] that does this and sends techstaff a report. The repo is currently not available for everyone to see but I think that it should be eventually. In the mean time you can take a look at it on the front end nodes (/usr/local/slurm-tools/cluster_partition_usage.py).
  
-You'll see on the spreadsheet we take subtract this number from 100, ensure it's positive and call that "idleness".+ 
 +You'll see on the spreadsheet we take subtract ''%%percent used%%'' from 100, ensure it's positive and call that "idleness".
  
 Total amount of money contributed and "idleness" are the key factors in determining the priority of a groups 'contrib' partition. Total amount of money contributed and "idleness" are the key factors in determining the priority of a groups 'contrib' partition.
Line 112: Line 109:
  
  
-Note that the term "idleness" should not be taken literally. I don't know of way to actually calculate true idleness. I do believe that the current calculation reflects the intent of the term. +Note that the term "idleness" should not be taken literally. I don't know of way to actually calculate true idleness. I believe that the current calculation reflects the intended usage.
- +
- +
- +
-==== Sheet usage ==== +
-  - All donations will get logged into the spreadsheet under the 'log' sheet. +
-  - Hardware donation gets converted to USD by techstaff. A receipt of the purchase is good starting point. +
-  - donations get assigned a POSIX group. Group must have a primary contact, who then gets to set members for that group. +
-  - The group 'cs' is calculated on the spreadsheet but doesn't actually get any priority set. 'general' is for all CS users. It needs to be part of the total sum but is treated as a special case with 0 priority. +
-  - Red: Do not edit +
-  - Green: user input (This will be Techstaff 95% of the time) +
-  - `groups` sheet: +
-    * calculates contribution amount for use in `contrib-priority`. +
-    * tracks group name and primary owner +
-  - `contrib-priority` calculation references contrib amounts calculated in `groups`. +
- +
  
  
/var/lib/dokuwiki/data/attic/techstaff/aicluster-admin.1612985028.txt.gz · Last modified: 2021/02/10 13:23 by kauffman
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki