Differences

This shows you the differences between two versions of the page.

--- techstaff:slurm [2017/08/24 14:54] – [Notice] kauffman
+++ techstaff:slurm [2017/11/13 10:02] – kauffman
@@ Line 1: / Line 1: @@
 ===== Notice =====
-**All is back to normal. Please submit jobs from linux1,2,3. Email techstaff@cs.uchicago.edu if you find that something is amiss.**
+**2017-08-31**: Configuration change to allow allocation on CPUs and RAM. Please read the 'Default Quota' section under https://howto.cs.uchicago.edu/techstaff:slurm#usage
-The SLURM cluster will become unavailable starting 2017-08-22 for an upgrade. Normal service should resume on 2017-08-25. Please check back here for status updates.
-**2017-08-22 1800**: Main cluster upgraded. You can try to use it now but I can't guarantee that I won't kill your job tomorrow or Firday.
-**2017-08-23 1345**: GPU servers upgraded and added back to the cluster. They may be missing some software that was not automatable at previous time of installation. Send me an email if you find anything missing.
-We still run systems with Ubuntu 14.04 installed. As of right now these systems cannot submit jobs to the cluster. This is on purpose. The slurm version jump between 14.04 and 16.04 was so huge that this was unavoidable. This means you should prefer to use linux.cs.uchicago.edu or any CS machine that run Ubuntu 16.04.
-**2017-08-24**: Everything seems to be working as expected. Please start using the cluster again. Email techstaff@cs.uchicago.edu if something is wrong/unexpected/broken/etc.
 ====== Peanut Job Submission Cluster ======
@@ Line 40: / Line 31: @@
   * [[http://slurm.schedmd.com/tutorials.html|SLURM tutorial videos]]
   * [[https://computing.llnl.gov/linux/slurm/quickstart.html|LLNL quick start user guide]]
+  * [[http://research.computing.yale.edu/support/hpc/user-guide/slurm| Yale's User Guide]]
 ===== Infrastructure =====
@@ Line 49: / Line 39: @@
   * 64gb RAM
   * 2x 500GB SATA 7200RPM in RAID1
-To better manage the cluster we have virtualized the job submission nodes and give them all resources of the hardware. So, the actual resources you can consume on any one node is:
-  * 14 Cores, 14 threads
-  * 62GB RAM
 ==== Storage ====
@@ Line 121: / Line 107: @@
 ===== Usage =====
 Below are some common examples. You should consult the [[http://slurm.schedmd.com/documentation.html|documentation]] of SLURM if you need further assistance.
+=== Default Quotas ===
+By default we set a job to be run on one CPU and allocate 100MB of RAM. If you require more than that you should specify what you need. Using the following options will do: ''%%--mem-per-cpu%%'', ''%%--nodes%%'', ''%%--ntasks%%''.
 === Exclusive access to a node ===
@@ Line 130: / Line 119: @@
 === Sample script ===
 Make sure you create a directory in which to deposit the ''%%STDIN%%'', ''%%STDOUT%%'', ''%%STDERR%%'' files.
-   mkdir -p $HOME/slurm/slurm_out
+   mkdir -p $HOME/slurm/out
 <code>
@@ Line 137: / Line 126: @@
 #SBATCH --mail-user=cnetid@cs.uchicago.edu
 #SBATCH --mail-type=ALL
-#SBATCH --output=/home/cnetid/slurm/slurm_out/%j.%N.stdout
+#SBATCH --output=/home/cnetid/slurm/out/%j.%N.stdout
-#SBATCH --error=/home/cnetid/slurm/slurm_out/%j.%N.stderr
+#SBATCH --error=/home/cnetid/slurm/out/%j.%N.stderr
 #SBATCH --workdir=/home/cnetid/slurm
 #SBATCH --partition=debug
@@ Line 144: / Line 133: @@
 #SBATCH --nodes=1
 #SBATCH --ntasks=1
+#SBATCH --mem-per-cpu=500
 #SBATCH --time=15:00
@@ Line 197: / Line 187: @@
 user@host:~$ sinfo
 PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
-debug*       up      30:00      1   idle research2
+debug*       up       30:00      1   idle slurm1
-general      up 1-00:00:00      6   idle research[3-8]
+general      up 14-00:00:00      6   idle slurm[2-6,8]
+pascal       up  3-00:00:00      1   idle gpu2
+tesla        up  3-00:00:00      1   idle gpu1
 </code>
@@ Line 224: / Line 216: @@
    srun -p general --pty --cpus-per-task 1 --mem 500 -t 0-06:00 /bin/bash
 will start a command line shell (''%%/bin/bash%%'') on the 'general' queue with 500 MB of RAM for 6 hours; 1 core on 1 node is assumed as these parameters (''%%-n 1 -N 1%%'') were left out. When the interactive session starts, you will notice that you are no longer on a login node, but rather one of the compute nodes dedicated to this queue. The ''%%--pty%%'' option allows the session to act like a standard terminal.
-====== Job Scheduling ======
+====== Job Scheduling ======
 We use a [[http://slurm.schedmd.com/priority_multifactor.html|multifactor]] method of job scheduling. Job priority is assigned by a combination of fair-share, partition priority, and length of time a job has been sitting in the queue. The priority of the queue is the highest factor in the job priority calculation. For certain queues this will cause jobs on lower priority queues which overlap with that queue to be requeued. The second most important factor is fair-share score. You can find a description of how SLURM calculates Fair-share [[http://slurm.schedmd.com/priority_multifactor.html#fairshare|here]]. The third most important is how long you have been sitting in the queue. The longer your job sits in the queue the higher its priority grows. If everyone’s priority is equal then FIFO is the scheduling method. If you want to see what your current priority is just do ''%%sprio -j JOBID%%'' which will show you the calculation it does to figure out your job priority. If you do ''%%sshare -u USERNAME%%'' you can see your current fair-share and usage.((https://rc.fas.harvard.edu/resources/running-jobs))