`space` - Workstation for Space Physics Lab¶

Hardware Spec¶

Motherboard: ASUS TUF X299 MARK1
CPU: Intel® i9-10900K x 1EA
RAM: Samsung DDR4-21300 32GB x 4EA
SDD: 970 EVO Plus M.2 NVMe 500GB x 2EA
Graphics Card: GIGABYTE GeForce RTX 2060 SUPER WINDFORCE OC D6 8GB
Case: FRACTAL DESIGN Define XL R2 Black Pearl
PSU: FSP HYDRO G PRO 850W 80PLUS Gold Full Modular
CPU Cooler: NOCTUA NH-D15 CROMAX,BLACK

OS Installation and Post-installation Settings¶

The OS distribution to be installed is CentOS 8.

Download an iso image from CentOS Download.

Creating bootable USB stick out of an ISO installation image¶

On macOS, follow the steps at How To Make a Bootable USB Stick From an ISO File on an Apple Mac OX X.

Download the desired file.
Open the Terminal.

Convert the .iso file to .img using the convert option of hdiutil:

$ hdiutil convert -format UDRW -o /path/to/target.img /path/to/source.iso

OS X tends to put the .dmg ending on the output file automatically. Rename the file by typing:
```
$ mv /path/to/target.img.dmg /path/to/target.img
```
Run diskutil list to get the current list of devices.
Insert your flash media.
Run diskutil list again and determine the device node assigned to your flash media (e.g. /dev/disk2).
Unmount the flash media (replace N with the disk number from the last command - in the previous example, N would be 2):
```
$ diskutil unmountDisk /dev/diskN
```
Copy the installation image (replace /path/to/downloaded.img with the path where the image file is located; for example, ./ubuntu.img or ./ubuntu.dmg):
```
$ sudo dd if=/path/to/downloaded.img of=/dev/rdiskN bs=1m
```
Eject your flash media when the command completes.
```
$ diskutil eject /dev/diskN
```

OS Installation¶

USB-boot the computer.

For text-mode installation, add text to the end of the kernel argument.

Choose the Custom Operating System mode.

Swapfile¶

See How to Add Swap Space on CentOS 8.

SSH Settings¶

Generate public/private rsa key pair for root (and the admin users if any):

$ ssh-keygen -t rsa -b 4096 -C "root@space.kyungguk.com"

Copy appropriate public keys to ~/.ssh/authorized_keys and change the file permission:

$ chmod 0600 ~/.ssh/authorized_keys

Edit /etc/ssh/sshd_config to include

PermitRootLogin without-password
PubkeyAuthentication yes
PermitEmptyPasswords no
PasswordAuthentication no
ChallengeResponseAuthentication yes

Restart the sshd service

# systemctl restart sshd

2FA - Google Authenticator PAM¶

See How To Set Up Two factor (2FA) Authentication for SSH on CentOS / RHEL 8/7. Particularly, for the settings of SElinux, see Multifactor authentication with Google Authenticator.

To add one-time passcode authentication for the SSH login to the head node,

# dnf install google-authenticator qrencode

Edit /etc/pam.d/sshd to include (appearing as the first auth)

-auth      required     pam_google_authenticator.so secret=${HOME}/.ssh/.google_authenticator

The secret phrase points to the config file where the sshd daemon can freely modify.

Modify /etc/ssh/sshd_config to include

ChallengeResponseAuthentication yes

Execute google-authenticator as a user and follow the instructions.

Limit on Super User Priviliage¶

Limit on `sudoers`¶

For security reasion, it is recommented to hard-code the default editor for visudo.

Execute EDITOR=nano visudo and add

Defaults editor=/usr/bin/nano

Check if users in group wheel is allowed to run all commands

%wheel  ALL=(ALL)    ALL

Limit on `su`¶

To limit su to a user in the wheel group, edit /etc/pam.d/su and uncomment

auth            required        pam_wheel.so use_uid

Base System Package Update¶

Synchronize the clocks

$ systemctl status chronyd
$ chronyc tracking

Update all packages installed during OS installation.

# dnf update

Enable Extra Packages for Enterprise Linux (EPEL) Repository

# dnf install epel-release
# dnf update

Install a couple of extra handy packages

# dnf install htop screen neovim indent

Graphics Card Driver and CUDA Installation¶

Graphics Card¶

See How to install the NVIDIA drivers on CentOS 8.

Download the Nvidia driver package from nvidia.com/download (or from UNIX DRIVER ARCHIVE).
Next, install package prerequisites for a successful Nvidia driver compilation and installation
```
# dnf groupinstall "Development Tools"
# dnf install libglvnd-devel elfutils-libelf-devel
```

Permanently disable the nouveau driver by modifying the GRUB boot menu

# grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) nouveau.modeset=0"

Reboot your CentOS 8 Desktop
The Nvidia drivers must be installed while the Xorg server is stopped. Switch to the text mode if needed:
```
# systemctl set-default multi-user.target
```
Install the Nvidia driver by executing the following command via TTY console or remote SSH login
```
# bash NVIDIA-Linux-x86_64-*
```

When prompted answer YES to installation of NVIDIA's 32-bit compatibility libraries and NO to run the nvidia-xconfig utility to automatically update your X configuration file.

Reboot your system one more time

All done. The Nvidia driver should now be installed on your CentOS 8 Desktop. Reboot your system now, login and run nvidia-settings to further configure your Nvidia graphic card settings.

CUDA 10.2¶

See CUDA 10.2 : Install.

Download CUDA Repository RPM package from the site below and Install it.

[space@root ~]# dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
Adding repo from: http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

# install with enabling EPEL too
[space@root ~]# dnf --enablerepo=epel -y install cuda-10-1

[space@root ~]# vi /etc/profile.d/cuda-10_2.sh

# set the environment variables
export PATH=/usr/local/cuda-10.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

[space@root ~]# source /etc/profile.d/cuda-10_2.sh

[space@root ~]# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Verify installation with a common user to run sample program.

# copy samples
[kmin@space ~]$ cuda-install-samples-10.2.sh ./
Copying samples to ./NVIDIA_CUDA-10.2_Samples now...
Finished copying samples.

[kmin@space ~]$ cd ./NVIDIA_CUDA-10.2_Samples/1_Utilities/deviceQuery

# compile deviceQuery sample
[kmin@space deviceQuery]$ make

# run deviceQuery sample
[kmin@space deviceQuery]$ ./deviceQuery
./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce RTX 2060 SUPER"
CUDA Driver Version / Runtime Version          11.1 / 10.1
CUDA Capability Major/Minor version number:    7.5
Total amount of global memory:                 7979 MBytes (8366915584 bytes)
(34) Multiprocessors, ( 64) CUDA Cores/MP:     2176 CUDA Cores
GPU Max Clock rate:                            1680 MHz (1.68 GHz)
Memory Clock rate:                             7001 Mhz
Memory Bus Width:                              256-bit
L2 Cache Size:                                 4194304 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  1024
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch:                          2147483647 bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
Run time limit on kernels:                     No
Integrated GPU sharing Host Memory:            No
Support host page-locked memory mapping:       Yes
Alignment requirement for Surfaces:            Yes
Device has ECC support:                        Disabled
Device supports Unified Addressing (UVA):      Yes
Device supports Compute Preemption:            Yes
Supports Cooperative Kernel Launch:            Yes
Supports MultiDevice Co-op Kernel Launch:      Yes
Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.1, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS

# try to run bandwidthTest sample
[kmin@space deviceQuery]$ cd ~/NVIDIA_CUDA-10.2_Samples/1_Utilities/bandwidthTest

[kmin@space bandwidthTest]$ make

[kmin@space bandwidthTest]$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...

Device 0: GeForce RTX 2060 SUPER
Quick Mode

Host to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes)    Bandwidth(GB/s)
32000000            13.0

Device to Host Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes)    Bandwidth(GB/s)
32000000            13.1

Device to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes)    Bandwidth(GB/s)
32000000            380.1

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Desktop Environment¶

Xfce Desktop¶

See Desktop Environment : Xfce Desktop.

Install Xfce Desktop Environment on this example.
```
[root@space ~]# dnf groupinstall Xfce
```
After installing environment, to start Desktop session on CUI, re-login with a common user and run like follows.
```
[kmin@space ~]$ echo "exec /usr/bin/xfce4-session" >> ~/.xinitrc
[kmin@space ~]$ startx
```
Or, if you would like to change your System to Graphical Login as default,
```
[root@space ~]# systemctl set-default graphical.target
```

and reboot.

Install Some Useful Modules¶

[root@space ~]# dnf install firefox
[root@space ~]# dnf module install gimp go-toolset inkscape llvm-toolset rust-toolset subversion

LVM Partition¶

See LFCS: How to Manage and Create LVM Using vgcreate, lvcreate and lvextend Commands – Part 11.

To create physical volumes on top of /dev/sda, do:

# pvcreate /dev/sda

List the newly created PVs

# pvdisplay

To create a volume group named vg00 using /dev/sda

# vgcreate lacie_16T /dev/sda

To view information about this volume group

# vgdisplay lacie_16T

To creating logical volumes

# lvcreate -n home2 -l 100%FREE lacie_16T

To view the list of LVs and detailed information with

# lvdisplay

Before each logical volume can be used, we need to create a filesystem on top of it.

# mkfs.xfs /dev/lacie_16T/home2

To find UUID

# blkid /dev/lacie_16T/home2

Create mount points

# mkdir /home2

Insert the corresponding entries in /etc/fstab (make sure to use the UUIDs obtained before)

UUID=b85df913-580f-461c-844f-546d8cde4646 /home2    xfs defaults 0 0

Disk Quota¶

See 3.3. XFS QUOTA MANAGEMENT, How to Setup Disk Quota on XFS File System in Linux Servers, and Using xfs project quotas to limit capacity within a subdirectory.

To enable User, Group, and Project quota on /home and /home2, edit the /etc/fstab to include thequota option:

...   /home    xfs    defaults,uquota,gquota,pquota   0 0
...   /scratch    xfs    defaults,uquota,gquota,pquota   0 0

Reboot and verify whether quota is enabled:

$ mount

To print disk quota,

$ xfs_quota -x -c 'report -hbi' /home
$ xfs_quota -x -c 'report -hbi' /home2

User Quota¶

To set disk (block) limit on a user named kmin on home directory,

$ xfs_quota -x -c 'limit -u bsoft=5g bhard=7g kmin' /home
$ xfs_quota -x -c 'limit -u bsoft=1t bhard=1200g kmin' /home2

In above command, bsoft is block soft limit in MBs and bhard is block hard limit in MBs, limit is a keyword to implement disk or file limit on a file system for a specific user.

To set file (inode) limit,

$ xfs_quota -x -c 'limit -u isoft=400 ihard=500 kmin' /home
$ xfs_quota -x -c 'limit -u isoft=40000 ihard=50000 kmin' /home2

In above command, isoft is inode or file soft limit and ihard is inode or file hard limit.

Both commands can be combined

$ xfs_quota -x -c 'limit -u bsoft=5g bhard=7g isoft=400 ihard=500 kmin' /home
$ xfs_quota -x -c 'limit -u bsoft=1t bhard=1200g isoft=40000 ihard=50000 kmin' /home2

Verify block and inode limits:

$ xfs_quota -x -c "report -bih" /home
$ xfs_quota -x -c "report -bih" /home2

Group Quota¶

To configure disk and file quota on engineering group,

$ xfs_quota -x -c 'limit -g bsoft=6144m bhard=8192m isoft=1000 ihard=1200 engineering' /home2

Verify the Quota details for group engineering

$ xfs_quota -x -c "report -gbih" /home2

Project Quota¶

Project quota is good for enforcing directory-based quota.
First, edit /etc/projects

2000:/home/kmin
102000:/home/scratch/kmin

And, edit /etc/projid to map the id to a name

kmin/home:2000
kmin/scratch:102000

Initialize project directories

$ xfs_quota -x -c 'project -s kmin/home' /home
$ xfs_quota -x -c 'project -s kmin/scratch' /home

Implement quotas

$ xfs_quota -x -c 'limit -p bsoft=8g bhard=10g kmin/home' /home
$ xfs_quota -x -c 'limit -p bsoft=800g bhard=1000g kmin/scratch' /home

Verify quota details

$ xfs_quota -x -c 'report -bih' /home

space - Workstation for Space Physics Lab¶