How do I?

User Tools

Site Tools

You are here: start » techstaff » slurm
techstaff:slurm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Next revisionBoth sides next revision
techstaff:slurm [2015/12/04 13:58] – kauffmantechstaff:slurm [2015/12/29 15:51] – [Partitions / Queues] kauffman
Line 1: Line 1:
-====== DRAFT | Peanut Job Submission Cluster | DRAFT ======+====== DRAFT | Peanut Job Submission Cluster ======
  
 We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might have been rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3). We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might have been rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3).
Line 37: Line 37:
  
 ==== Storage ==== ==== Storage ====
-Shared scratch storage is being planned, but not yet available. Techstaff hopes to have this done in time for the winter quarter.+There is slow scratch space mounted to ''%%/scratch%%''. It is a ZFS pool consisting of 10x 2TB 7200RPM SAS drives connected via a LSI 9211-8i and is made up of 5 mirrored VDEVs, which is similar to a RAID10. The servers uplink is 1G ethernet. 
  
 +  * Files older than 90 days will be deleted automatically.
 +  * Scratch space is shared by all users.
 ==== Utilization Dashboard ==== ==== Utilization Dashboard ====
 Sometimes it is useful to see how much of the cluster is utilized. You can do that via the following URL: http://peanut.cs.uchicago.edu Sometimes it is useful to see how much of the cluster is utilized. You can do that via the following URL: http://peanut.cs.uchicago.edu
  
 ==== Partitions / Queues ==== ==== Partitions / Queues ====
-As of December, 2015 we have 3 partitions in our cluster.+To find out what partitions we offer, checkout the [[techstaff:slurm#sinfo|sinfo]] command. 
 + 
 +As of December, 2015 we have will have at least 2 partitions in our cluster; 'debug' and 'general'. An other partition is not guarenteed and will server a specific purpose.
  
 ^ Partition Name ^ Description ^ ^ Partition Name ^ Description ^
 | **debug** | The partition your job will be submitted to if none is specified. The purpose of this partition is to make sure your code is running as it should before submitting a long running job to the general queue. | | **debug** | The partition your job will be submitted to if none is specified. The purpose of this partition is to make sure your code is running as it should before submitting a long running job to the general queue. |
 | **general** | All jobs that have been thoroughly tested can be submitted here. This partition will have access to more nodes and will process most of the jobs. If you need to use the ''%%--exclusive%%'' flag it should be done here.| | **general** | All jobs that have been thoroughly tested can be submitted here. This partition will have access to more nodes and will process most of the jobs. If you need to use the ''%%--exclusive%%'' flag it should be done here.|
-| **hardware** | The purpose of this partition is to set aside nodes that do not use the virtualization layer. This is important if you need true hardware access. | 
  
  
Line 145: Line 148:
  
  
-===== Monitoring Jobs =====+====== Monitoring Jobs ======
  
 ''%%squeue%%'' and ''%%sacct%%'' are two different commands that allow you to monitor job activity in SLURM. ''%%squeue%%'' is the primary and most accurate monitoring tool since it queries the SLURM controller directly. ''%%sacct%%'' gives you similar information for running jobs, and can also report on previously finished jobs, but because it accesses the SLURM database, there are some circumstances when the information is not in sync with squeue. ''%%squeue%%'' and ''%%sacct%%'' are two different commands that allow you to monitor job activity in SLURM. ''%%squeue%%'' is the primary and most accurate monitoring tool since it queries the SLURM controller directly. ''%%sacct%%'' gives you similar information for running jobs, and can also report on previously finished jobs, but because it accesses the SLURM database, there are some circumstances when the information is not in sync with squeue.
Line 156: Line 159:
  
  
-===== Interactive Jobs =====+====== Interactive Jobs ======
  
 Though batch submission is the best way to take full advantage of the compute power in the job submission cluster, foreground, interactive jobs can also be run. Though batch submission is the best way to take full advantage of the compute power in the job submission cluster, foreground, interactive jobs can also be run.
Line 168: Line 171:
 will start a command line shell (''%%/bin/bash%%'') on the 'general' queue with 500 MB of RAM for 6 hours; 1 core on 1 node is assumed as these parameters (''%%-n 1 -N 1%%'') were left out. When the interactive session starts, you will notice that you are no longer on a login node, but rather one of the compute nodes dedicated to this queue. The ''%%--pty%%'' option allows the session to act like a standard terminal. will start a command line shell (''%%/bin/bash%%'') on the 'general' queue with 500 MB of RAM for 6 hours; 1 core on 1 node is assumed as these parameters (''%%-n 1 -N 1%%'') were left out. When the interactive session starts, you will notice that you are no longer on a login node, but rather one of the compute nodes dedicated to this queue. The ''%%--pty%%'' option allows the session to act like a standard terminal.
  
-===== Job Scheduling =====+====== Job Scheduling ======
  
 We use a [[http://slurm.schedmd.com/priority_multifactor.html|multifactor]] method of job scheduling. Job priority is assigned by a combination of fair-share, partition priority, and length of time a job has been sitting in the queue. The priority of the queue is the highest factor in the job priority calculation. For certain queues this will cause jobs on lower priority queues which overlap with that queue to be requeued. The second most important factor is fair-share score. You can find a description of how SLURM calculates Fair-share [[http://slurm.schedmd.com/priority_multifactor.html#fairshare|here]]. The third most important is how long you have been sitting in the queue. The longer your job sits in the queue the higher its priority grows. If everyone’s priority is equal then FIFO is the scheduling method. If you want to see what your current priority is just do ''%%sprio -j JOBID%%'' which will show you the calculation it does to figure out your job priority. If you do ''%%sshare -u USERNAME%%'' you can see your current fair-share and usage.((https://rc.fas.harvard.edu/resources/running-jobs)) We use a [[http://slurm.schedmd.com/priority_multifactor.html|multifactor]] method of job scheduling. Job priority is assigned by a combination of fair-share, partition priority, and length of time a job has been sitting in the queue. The priority of the queue is the highest factor in the job priority calculation. For certain queues this will cause jobs on lower priority queues which overlap with that queue to be requeued. The second most important factor is fair-share score. You can find a description of how SLURM calculates Fair-share [[http://slurm.schedmd.com/priority_multifactor.html#fairshare|here]]. The third most important is how long you have been sitting in the queue. The longer your job sits in the queue the higher its priority grows. If everyone’s priority is equal then FIFO is the scheduling method. If you want to see what your current priority is just do ''%%sprio -j JOBID%%'' which will show you the calculation it does to figure out your job priority. If you do ''%%sshare -u USERNAME%%'' you can see your current fair-share and usage.((https://rc.fas.harvard.edu/resources/running-jobs))
Line 175: Line 178:
  
  
-===== Common Issues =====+====== Common Issues ======
  
 ^Error ^What does it mean? ^ ^Error ^What does it mean? ^
Line 182: Line 185:
 | JOB <jobid> CANCELLED AT <time> DUE TO NODE FAILURE | There can be many reasons for this message, but most often it means that the node your job was set to run on can no longer be contacted by the the SLURM controller.| | JOB <jobid> CANCELLED AT <time> DUE TO NODE FAILURE | There can be many reasons for this message, but most often it means that the node your job was set to run on can no longer be contacted by the the SLURM controller.|
 | error: Unable to allocate resources: More processors requested than permitted | It usually has **nothing** to do with priviledges you may or may not have. Rather, it usually means that you have allocated more processors than one compute node actually has. | | error: Unable to allocate resources: More processors requested than permitted | It usually has **nothing** to do with priviledges you may or may not have. Rather, it usually means that you have allocated more processors than one compute node actually has. |
 +
 +====== More ======
 +If you feel this documentation is lacking in some way please let techstaff know. Email(techstaff@cs.uchicago.edu), call(773-702-1031), or stop by our office (Ryerson 154).
/var/lib/dokuwiki/data/pages/techstaff/slurm.txt · Last modified: 2021/01/06 16:13 by kauffman
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki