How do I?

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== AI Cluster Policy Description ======
===== TODO =====
  - There are multiple methods used to calculate priority reflected on the spreadsheet.
  - Double check the math for correctness.
  - Does the math reflect our intent? I think it does.
  - Multiple methods in calculating priority are reflected (blue to purple cells: ''%%har_priority%%'', ''%%normalized_priority%%'', ''%%weighted_normalized_priority%%''
  - Choose one calculation to use. This decision doesn't prevent us from changing it later if we find another calculation works better.
  - If you have a suggestion: Please show us the work in a new column or by cloning the sheet.

===== Contribution Tracking and Priority Calculation Tool =====

[[https://docs.google.com/spreadsheets/d/15o3jZOVqU84hMevIKnLKj8DhFYaQUdPdOYE1_itucqk|AI Cluster Tracking and Priority calculation spreadsheet]]

AI Cluster committee members have access.

==== Sheet usage ====
  * Red: Do not edit
  * Green: user input (This will be Techstaff 95% of the time)
  * `groups` sheet:
    * contributions get assigned a POSIX group. Group must have a primary contact, who then gets to set members for that group.
    * calculates contribution amount for use in `contrib-priority`.
    * tracks group name and primary owner
  * `log` sheet
    * All contributions will get entered here.
    * Hardware contribution gets converted to USD by techstaff. A receipt of the purchase is good starting point.
  * The group 'cs' is calculated on the spreadsheet but doesn't actually get any priority set. 'general' is for all CS users. It needs to be part of the total sum but is treated as a special case with 0 priority.
  * `contrib-priority` calculation references contrib amounts calculated in `groups`.




===== Understanding Slurm Fairshare and Priority/Multifactor =====
Slurm comes with built in tools to calculation fair share priorities without anyone needing to do anything special. The cluster uses a [[https://slurm.schedmd.com/fair_tree.html|fair share algorithm]] with [[https://slurm.schedmd.com/priority_multifactor.html|multiple factors]] to adjust job priorities.

==== How Slurm calculates Job priority ====
Generally Slurm will use this formula to determine a jobs priority.
<code>
Job_priority =
	site_factor +
	(PriorityWeightAge) * (age_factor) +
	(PriorityWeightAssoc) * (assoc_factor) +
	(PriorityWeightFairshare) * (fair-share_factor) +
	(PriorityWeightJobSize) * (job_size_factor) +
	(PriorityWeightPartition) * (partition_factor) +
	(PriorityWeightQOS) * (QOS_factor) +
	SUM(TRES_weight_cpu * TRES_factor_cpu,
	    TRES_weight_<type> * TRES_factor_<type>,
	    ...)
	- nice_factor
</code>

The factors on the left that start with ''%%PriorityWeight*%%'' can and are set in the Slurm cluster configuration file (found on all nodes at /etc/slurm-llnl/slurm.conf). The factors on the right are calculated based previous job submissions for that particular user.

<code>
fe01:~$ cat /etc/slurm-llnl/slurm.conf |grep "^Priority"
PriorityType=priority/multifactor
PriorityDecayHalfLife=08:00:00
PriorityMaxAge=5-0
PriorityWeightFairshare=500000
PriorityWeightPartition=100000
PriorityWeightQOS=0
PriorityWeightJobSize=0
PriorityWeightAge=0
PriorityFavorSmall=YES
</code>
*Note that this example may not be up to date when you read this.

===== How we modify job priority to favor contributors =====

  * We adjust those priorities with partitions for those who have donated either monetarily or with hardware. Hardware donations get converted a monetary value when logged on the spreadsheet.
  * Every contribution gets assigned a POSIX group, hereon referred to as ''%%$group%%''.


Here is a version of the partition configuration as it stands now (2021-02-10).
<code>
PartitionName=general Nodes=a[001-008]
#PartitionName=cdac-own Nodes=a[005-008] AllowGroups=cdac Priority=100
PartitionName=cdac-contrib Nodes=a[001-008] AllowGroups=cdac Priority=5
</code>

^Partition^Description^Priority^
|general| For all users| 0 |
|${group}-own | Machines $group has donated. Enabled when asked. | 100 |
|${group}-contrib | A method to give slightly higher job priority to groups who have donated but do not own machines.| Variable based on spreadsheet calculation. |

The key thing to notice before you continue reading is that nodes can be added to multiple partitions. ''%%general%%'' and ''%%cdac-contrib%%'' can submit to all nodes but with different priorities.


==== Calculating -contrib partition usage ====

We do the following calculation to determine the ''%%*-contrib%%'' partitions usage over the past 30 days in comparison to total cluster usage.

<code>
partition usage total time in seconds for 30 days
------------------------------------------------------ = percent used
all partition usage total time in seconds for 30 days
</code>

The percent will end up as an integer.

There is a [[https://vcs.cs.uchicago.edu/kauffman/slurm-tools/blob/master/cluster_partition_usage.py|python script]] that does this and sends techstaff a report. The repo is currently not available for everyone to see but I think that it should be eventually. In the mean time you can take a look at it on the front end nodes (/usr/local/slurm-tools/cluster_partition_usage.py).


You'll see on the spreadsheet we take subtract ''%%percent used%%'' from 100, ensure it's positive and call that "idleness".

Total amount of money contributed and "idleness" are the key factors in determining the priority of a groups 'contrib' partition.

This calculation will be run once a month and the relevant groups ${group}-contrib priority updated to reflect past months usage.


Note that the term "idleness" should not be taken literally. I don't know of way to actually calculate true idleness. I believe that the current calculation reflects the intended usage.




====== AI Cluster Admin ======

===== TODO =====

Since I'm still working on it, I don't guarantee any uptime yet. Mainly I need to make sure TRES tracking is working like we want. This will involve restarting slurmd and slurmctld which will kill running jobs.


  * <del>generate report of storage usage</del>
  * <del>groups (Slurm 'Accounts') created for PI's.</del>
    * <del>e.g. ericj_group: ericj, user1, user1, etc</del>
  * <del>grab QOS data from somewhere (gsheet or some kind of DB)</del>
  * <del>Properly deploy sync script</del>
    * <del>Systemd unit</del>
    * <del>main loop</del>
  * <del>research on slurm plugin to force GRES selection on job submit. Might be able to use: </del>
    * <del>SallocDefaultCommand</del>
    * <del>Otherwise look for 'AccountingStorageTRES' and 'JobSubmitPlugins' and  /etc/slurm-llnl/job_submit.lua <= used to force user to specify '--gres'.</del>
    * <del>jobs that do not specify a specific gpu type (e.g. gpu:rtx8000 or gpu:rtx2080ti) could be counted against either one but not specifically the you actually used.</del>
    * <del>From 'AccountingStorageTRES' in slurm.conf: "Given a configuration of "AccountingStorageTRES=gres/gpu:tesla,gres/gpu:volta" Then "gres/gpu:tesla" and "gres/gpu:volta" will track jobs that explicitly request those GPU types. If a job requests GPUs, but does not explicitly specify the GPU type, then its resource allocation will be accounted for as either "gres/gpu:tesla" or "gres/gpu:volta", although the accounting may not match the actual GPU type allocated to the job and the GPUs allocated to the job could be heterogeneous. In an environment containing various GPU types, use of a job_submit plugin may be desired in order to force jobs to explicitly specify some GPU type."</del>
  * <del>ganglia for Slurm: http://ai-mgmt2.ai.cs.uchicago.edu</del>
    * <del>figure why summary view is no longer a thing.</del>
  * <del>update 'coolgpus'. Lose VTs when this is running.</del>
    * <del>coolgpus: sets fan speeds of all gpus in system.</del>
    * <del>Goal is to statically set fan speeds to 80%. The only way to do this is with fake Xservers... but that means you lose all the VTs. Is this a compromise I'm willing to make?</del> It is.
  * home directory
    * <del>setup backups for home dirs</del>
    * <del>default quota</del>
    * <del>userland tool to check quota</del>
    * <del>home directory usage report</del>
  * monitoring
    * <del>basic node monitor</del>
    * nfs or bandwidth monitoring
    * gpu
  * sync script
    * <del>fix bug to ensure accounts and users are created</del>