Differences

This shows you the differences between two versions of the page.

--- techstaff:slurm [2015/12/29 15:14] – kauffman
+++ techstaff:slurm [2017/08/31 15:12] – [Usage] kauffman
@@ Line 1: / Line 1: @@
-====== DRAFT | Peanut Job Submission Cluster ======
+===== Notice =====
+**All is back to normal. Please submit jobs from linux1,2,3. Email techstaff@cs.uchicago.edu if you find that something is amiss.**
-We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might have been rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3).
+The SLURM cluster will become unavailable starting 2017-08-22 for an upgrade. Normal service should resume on 2017-08-25. Please check back here for status updates.
+**2017-08-22 1800**: Main cluster upgraded. You can try to use it now but I can't guarantee that I won't kill your job tomorrow or Firday.
+**2017-08-23 1345**: GPU servers upgraded and added back to the cluster. They may be missing some software that was not automatable at previous time of installation. Send me an email if you find anything missing.
+We still run systems with Ubuntu 14.04 installed. As of right now these systems cannot submit jobs to the cluster. This is on purpose. The slurm version jump between 14.04 and 16.04 was so huge that this was unavoidable. This means you should prefer to use linux.cs.uchicago.edu or any CS machine that run Ubuntu 16.04.
+**2017-08-24**: Everything seems to be working as expected. Please start using the cluster again. Email techstaff@cs.uchicago.edu if something is wrong/unexpected/broken/etc.
+====== Peanut Job Submission Cluster ======
+We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might be rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3).
 For job submission we will be using a piece of software called [[http://slurm.schedmd.com|SLURM]]. Simply put, SLURM is a queue management system and stands for **S**imple **L**inux **U**tility for **R**esource **M**anagement; it was developed at the Lawrence Livermore National Lab. It currently supports some of the largest compute clusters in the world. The best description of SLURM can be found on its homepage:
@@ Line 8: / Line 20: @@
 SLURM is similar to most other queue systems in that you write a batch script, then submit it to the queue manager. The queue manager schedules your job to run on the queue (or partition in SLURM parlance) that you designate. Below is an outline of how to submit jobs to SLURM, how SLURM decides when to schedule your job, and how to monitor progress.
+===== Where to begin =====
+SLURM is a set of command line utilities that can be accessed via the command line from **most** any computer science system you can login to. Using our main shell servers (linux.cs.uchicago.edu) is expected to be our most common use case, so you should start there.
+  ssh user@linux.cs.uchicago.edu
 ===== Documentation =====
@@ Line 31: / Line 49: @@
   * 64gb RAM
   * 2x 500GB SATA 7200RPM in RAID1
-To better manage the cluster we have virtualized the job submission nodes and give them all resources of the hardware. So, the actual resources you can consume on any one node is:
-  * 14 Cores, 14 threads
-  * 62GB RAM
 ==== Storage ====
 There is slow scratch space mounted to ''%%/scratch%%''. It is a ZFS pool consisting of 10x 2TB 7200RPM SAS drives connected via a LSI 9211-8i and is made up of 5 mirrored VDEVs, which is similar to a RAID10. The servers uplink is 1G ethernet.
+  * Files older than 90 days will be deleted automatically.
+  * Scratch space is shared by all users.
+=== Access ===
+Scratch space is only mounted on nodes associated with the cluster. If you want to be able to transfer files to the scratch space you will want to run an [[techstaff:slurm#interactive_jobs|interactive shell]]. Now you will be able to use standard tools such as ''%%scp%%'' or ''%%rsync%%'' to transfer files.
+  - You should only do a file transfer via the debug partition: ''%% srun -p debug --pty --mem 500 /bin/bash%%''
+  - Now you can create a directory of your own: ''%%mkdir /scratch/$USER%%'' You should store any files you create in this directory.
+== Example ==
+Request interactive shell
+<code>user@csilcomputer:~$ srun --pty --mem 500 /bin/bash </code>
+Change into my scratch directory:
+<code>user@research2:~$ cd /scratch/user/</code>
+Get the files I need:
+<code>
+user@research2:/scratch/user$ scp user@csilcomputer:~/foo .
+foo                         100%  103KB 102.7KB/s   00:00
+</code>
+Check that the file now exists:
+<code>
+user@research2:/scratch/user$ ls -l foo
+-rw------- 1 user user 105121 Dec 29  2015 foo
+</code>
+I can now exit my interactive shell.
+== Performance is slow ==
+This is expected. The maximum speed this server will ever be able to achieve is 1Gb/s because of its single 1G ethernet uplink. If this cluster gains in popularity we plan on upgrading the network and storage server.
 ==== Utilization Dashboard ====
 Sometimes it is useful to see how much of the cluster is utilized. You can do that via the following URL: http://peanut.cs.uchicago.edu
 ==== Partitions / Queues ====
-As of December, 2015 we have will have at least 2 partitions in our cluster; 'debug' and 'general'. An other partition is not guarenteed and will server a specific purpose.
+To find out what partitions we offer, checkout the [[techstaff:slurm#sinfo|sinfo]] command.
+As of December, 2015 we have will have at least 2 partitions in our cluster; 'debug' and 'general'.
 ^ Partition Name ^ Description ^
 | **debug** | The partition your job will be submitted to if none is specified. The purpose of this partition is to make sure your code is running as it should before submitting a long running job to the general queue. |
 | **general** | All jobs that have been thoroughly tested can be submitted here. This partition will have access to more nodes and will process most of the jobs. If you need to use the ''%%--exclusive%%'' flag it should be done here.|
-| **hardware** | The purpose of this partition is to set aside nodes that do not use the virtualization layer. This is important if you need true hardware access. |
+| **gpu** | Contains servers with graphics cards. As of May 2016 there is only one node containing a Tesla M2090. You will be forced to use this server exclusively for now. Please keep your time in interactive mode to a minimum.|
 ====== Job Submission ======
@@ Line 70: / Line 117: @@
 ===== Usage =====
 Below are some common examples. You should consult the [[http://slurm.schedmd.com/documentation.html|documentation]] of SLURM if you need further assistance.
+=== Default Quotas ===
+By default we set a job to be run on one CPU and allocate 100MB of RAM. If you require more than that you should specify what you need. Using the following options will do: ''%%--mem-per-cpu%%'', ''%%--nodes%%'', ''%%--ntasks%%''.
 === Exclusive access to a node ===
@@ Line 79: / Line 129: @@
 === Sample script ===
 Make sure you create a directory in which to deposit the ''%%STDIN%%'', ''%%STDOUT%%'', ''%%STDERR%%'' files.
-   mkdir -p $HOME/slurm/slurm_out
+   mkdir -p $HOME/slurm/out
 <code>
 #!/bin/bash
 #
-#SBATCH --mail-user=user@cs.uchicago.edu
+#SBATCH --mail-user=cnetid@cs.uchicago.edu
 #SBATCH --mail-type=ALL
-#SBATCH --output=/home/user/slurm/slurm_out/%j.%N.stdout
+#SBATCH --output=/home/cnetid/slurm/out/%j.%N.stdout
-#SBATCH --error=/home/user/slurm/slurm_out/%j.%N.stderr
+#SBATCH --error=/home/cnetid/slurm/out/%j.%N.stderr
-#SBATCH --workdir=/home/user/slurm
+#SBATCH --workdir=/home/cnetid/slurm
 #SBATCH --partition=debug
 #SBATCH --job-name=check_hostname_of_node
 #SBATCH --nodes=1
 #SBATCH --ntasks=1
+#SBATCH --mem-per-cpu=500
 #SBATCH --time=15:00
@@ Line 101: / Line 152: @@
   man sbatch
-Make sure to replace all instances of the word ''%%user%%'' with your CNETID.
+Make sure to replace all instances of the word ''%%cnetid%%'' with your CNETID.
+=== Submitting job script ===
+Using the above example you will want to place your tested code into a file. 'hostname.job' is the file name in this example.
+<code>
+sbatch hostname.job
+</code>
+You can then check the status via squeue or see the output in the output directory '$HOME/slurm/slurm_out'.
 ==== srun ====
 Used to submit a job to the cluster that doesn't necessarily need a script.
@@ Line 140: / Line 198: @@
 PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
 debug*       up      30:00      1   idle research2
-general      up   infinite      5   idle research[3-7]
+general      up 1-00:00:00      6   idle research[3-8]
-hardware     up    1:00:00      1   idle research8
 </code>
@@ Line 165: / Line 222: @@
 This command:
-   srun -p general --pty --mem 500 -t 0-06:00 /bin/bash
+   srun -p general --pty --cpus-per-task 1 --mem 500 -t 0-06:00 /bin/bash
 will start a command line shell (''%%/bin/bash%%'') on the 'general' queue with 500 MB of RAM for 6 hours; 1 core on 1 node is assumed as these parameters (''%%-n 1 -N 1%%'') were left out. When the interactive session starts, you will notice that you are no longer on a login node, but rather one of the compute nodes dedicated to this queue. The ''%%--pty%%'' option allows the session to act like a standard terminal.
 ====== Job Scheduling ======
@@ Line 183: / Line 239: @@
 | error: Unable to allocate resources: More processors requested than permitted | It usually has **nothing** to do with priviledges you may or may not have. Rather, it usually means that you have allocated more processors than one compute node actually has. |
+====== Using the GPU ======
+===== Paths =====
+You will need to add the following to your $PATH and $LD_LIBRARY_PATH.
+  export PATH=$PATH:/usr/local/cuda/bin
+  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/usr/local/cuda/lib
+===== Example =====
+This sbatch script will get device information from the installed Tesla gpu.
+<code>
+#!/bin/bash
+#
+#SBATCH --mail-user=cnetid@cs.uchicago.edu
+#SBATCH --mail-type=ALL
+#SBATCH --output=/home/cnetid/slurm/slurm_out/%j.%N.stdout
+#SBATCH --error=/home/cnetid/slurm/slurm_out/%j.%N.stderr
+#SBATCH --workdir=/home/cnetid/slurm
+#SBATCH --partition=gpu
+#SBATCH --job-name=get_tesla_info
+export PATH=$PATH:/usr/local/cuda/bin
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/usr/local/cuda/lib
+cat << EOF > /tmp/getinfo.cu
+#include <stdio.h>
+int main() {
+  int nDevices;
+  cudaGetDeviceCount(&nDevices);
+  for (int i = 0; i < nDevices; i++) {
+    cudaDeviceProp prop;
+    cudaGetDeviceProperties(&prop, i);
+    printf("Device Number: %d\n", i);
+    printf("  Device name: %s\n", prop.name);
+    printf("  Memory Clock Rate (KHz): %d\n",
+           prop.memoryClockRate);
+    printf("  Memory Bus Width (bits): %d\n",
+           prop.memoryBusWidth);
+    printf("  Peak Memory Bandwidth (GB/s): %f\n\n",
+.0*prop.memoryClockRate*(prop.memoryBusWidth/8)/1.0e6);
+  }
+}
+EOF
+/usr/local/cuda/bin/nvcc /tmp/getinfo.cu -o /tmp/a.out
+/tmp/a.out
+rm /tmp/a.out
+rm /tmp/getinfo.cu
+</code>
+==== Output ====
+STDOUT will look something like this:
+<code>
+cnetid@linux1:~$ cat $HOME/slurm/slurm_out/12567.gpu1.stdout
+Device Number: 0
+  Device name: Tesla M2090
+  Memory Clock Rate (KHz): 1848000
+  Memory Bus Width (bits): 384
+  Peak Memory Bandwidth (GB/s): 177.408000
+</code>
+STDERR should be blank.
 ====== More ======
-If you feel this documentation is lacking in some way please let techstaff know. Email(techstaff@cs.uchicago.edu), call(773-702-1031), or stop by our office (Ryerson 154).
+If you feel this documentation is lacking in some way please let techstaff know. Email [[techstaff@cs.uchicago.edu]], call (773-702-1031), or stop by our office (Ryerson 154).