Skip to content

HPC - Admin Guide

Preface

This document describes the cluster administration guide.

It consists of two parts:

  1. Cluster user accounting and frontend resource allocation, and
  2. Slurm user accounting and job scheduling/management.

Contents

Cluster User Accounting

All users are assigned a UID greater than or equal to 2000 and the system-default group (users for CentOS). In addition, they are assigned one or more groups that reflect the Slurm account names. These groups are also assigned a GID greater than or equal to 2000.

NOTE: Slurm accounting is limited to lower case user and group names.

Creating a user account on the cluster consists of

  1. Adding a user on the head node and creating a scratch directory;
  2. Configuring CGroup to limit user's resources on the head node;
  3. Setting disk quota on the /home and /scratch partitions;
  4. Propagating /etc/passwd and /etc/group from the head node to all other nodes; and
  5. Setting the default password and google-authenticator.

Deleting an existing user account involves

  1. Purging the CGroup configs on the head node;
  2. Resetting user's disk quota;
  3. Purging user's scratch directory;
  4. Purging the user record from the head node; and
  5. Propagating /etc/passwd and /etc/group from the head node to all other nodes.

The action below only applies to the head node.

Adding/Deleting a user

- Adding a user

To add a new user named $UNAME with $USRID (>= 2000) and with no group assigned (other than the system default),

$ sudo useradd -u $USRID -G research -N -m -c "comment goes here" -s /bin/zsh $UNAME

and create a scratch directory and set the owner and the permission:

$ sudo mkdir /scratch/$UNAME
$ sudo chown $UNAME: /scratch/$UNAME
$ sudo chmod 0700 /scratch/$UNAME

Propagate /etc/passwd and /etc/group from the head node to all other nodes.

$ for i in {1..10}; ssh node$i "useradd -u $USRID -G research -N -M -c 'comment goes here' $UNAME"; done

- Deleting a user

To delete an existing user $UNAME,

$ sudo userdel -r $UNAME

which will also remove the home and mail directories.

Finally, delete the scratch directory

$ sudo rm -rv /scratch/$UNAME/

Propagate /etc/passwd and /etc/group from the head node to all other nodes.

$ for i in {1..10}; ssh node$i "userdel $UNAME"; done

User slice CGroup

See Resource Management Guide and RHEL7: How to get started with CGroups.

Resource limits for normal users are set as follows:

  • CPUQuota=200% (equivalent to 2 threads)
  • MemoryLimit=4294967296 (4GB)
  • TasksMax=100

See below about how to request more resources for job preparation. TODO: Add a link.

- On user creation

Create a user slice CGroup config directory (for a given $USRID)

$ sudo mkdir /etc/systemd/system/user-$USRID.slice.d

Edit /etc/systemd/system/user-$USRID.slice.d/00-accounting.conf to include

[Slice]
CPUAccounting=true
MemoryAccounting=true
TasksAccounting=true
BlockIOAccounting=true

Edit /etc/systemd/system/user-$USRID.slice.d/99-limits.conf to include

[Slice]
CPUQuota=200%
MemoryLimit=4294967296
TasksMax=100

Reload systemd daemons

$ sudo systemctl daemon-reload

To check the settings

$ systemctl -t slice show user-$USRID.slice

- On user deletion

Delete /etc/systemd/system/user-$USRID.slice.d

$ sudo rm -rv /etc/systemd/system/user-$USRID.slice.d/

Reload systemd daemons

$ sudo systemctl daemon-reload

Disk quota

For the use of xfs_quota, see Disk Quota.

The soft and hard block limits (bsoft and bhard, respectively) are

  • 1000GB and 1200GB, respectively, on the /home partition
  • 5TB and 5200GB, respectively, on the /scratch partition

To set disk (block) limit on a user named $UNAME on the /home partition,

$ sudo xfs_quota -xc "limit -u bsoft=1000g bhard=1200g $UNAME" /home
$ sudo xfs_quota -xc "limit -u bsoft=5t bhard=5200g $UNAME" /scratch

Verify block and inode limits:

$ sudo xfs_quota -xc "report -bih" /home
$ sudo xfs_quota -xc "report -bih" /scratch

To reset the limits, set the limits with zeros as its argument, e.g.,

$ sudo xfs_quota -xc "limit -u bsoft=0 bhard=0 $UNAME" /home
$ sudo xfs_quota -xc "limit -u bsoft=0 bhard=0 $UNAME" /scratch

Default password and google-authenticator

All users accessing the frontend through SSH are two-factor authenticated (unless PublicKeyAuthentication is setup). For the clusterside, the google-authenticator PAM module is used for the one-time passcode authentication.

To generate the google-authenticator token, login as the user and execute

$ google-authenticator

Follow the instructions, while setting

  • Impose the rate limit,
  • Disallow reuse of the one-time code, and
  • Do not set the leeway.

Upon completion, the terminal will display a QR code (make sure than the terminal is sufficiently large) which will be relayed to the user for registration.

Google-authenticator Sample QR Code

Finally, set the default password using

$ sudo passwd $UNAME

Slurm User Accounting

Some references:

Slurm provides a very fine-grained control of user accounting and job scheduling.

sacctmgr is used to view or modify Slurm account information. The account information is maintained within a database with the interface being provided by slurmdbd (Slurm Database daemon). This database can serve as a central storehouse of user and computer information for multiple computers at a single site. Slurm account information is recorded based upon four parameters that form what is referred to as an association. These parameters are user, cluster, partition, and account. user is the login name. cluster is the name of a Slurm managed cluster as specified by the ClusterName parameter in the slurm.conf configuration file. partition is the name of a Slurm partition on that cluster. account is the bank account for a job. The intended mode of operation is to initiate the sacctmgr command, add, delete, modify, and/or list association records then commit the changes and exit.

scontrol

sinfo -al

sinfo -Nel