HPC - Remote Server Management

Preface

This document describes the remote server management setup.

It consists of two parts:

• Setting up a Raspberry Pi 4 to redirect all network traffic
• Server's BMC network setup

Contents

• Setup Raspberry Pi
• Raspberry Pi Router
• BMC Network Setup
• Yoctopuce's Environmental Sensors

---

Setup Raspberry Pi

The Raspberry Pi module that I chose is SeeedStudio's Mini Router/PC with Raspberry Pi Compute Module 4, Dual Gigabit Ethernet NICs, 4GB RAM/32GB eMMC. The online resources for this product are:

• Raspberry Pi Compute Module 4
• How to Build a Raspberry Pi Router - Step by Step Tutorial
• Getting Started with Dual Gigabit Ethernet Carrier Board for Raspberry Pi Compute Module 4
• Getting Started with OpenWrt on Dual Gigabit Ethernet Carrier Board for Raspberry Pi CM4 and ODYSSEY - X86J4125

Flashing AlmaLinux 8 Arm/AArch64

After flashing and testing various OSs, I chose AlmaLinux 8 Arm/AArch64 because it's a RHEL-base Linux which I am familiar with and it supports XRDP.

Flashing this OS was not very difficult. I followed AlmaLinux Official Raspberry Pi Image.

1. Install RPI Imager. This is also available from the Snap Store.

2. Since CM4 uses eMMC, follow the instruction to connect the module to a computer as an external storage.

3. Follow through the steps in AlmaLinux Raspberry Pi Quick Start. The default root password is almalinux. The two ethernet ports are recognized out of the box.

   • Step 1: Grab the image, verify the CHECKSUM and burn it to an SD card using RPi Image.
   • Step 2: Insert your SD Card into your Raspberry PI and boot!
   • Step 3: Login. The user is root password is almalinux.
   • Step 4: Resize your root filesystem by running rootfs-expand.

If you installed a GUI, and your screen has a black border around it, edit /boot/config.txt to add disable_overscan=1.

Setting up the OS

Once booting and ssh'ing are successful, the rest of the software installation and configuration is the same as setting up a usual Linux box.

1. Disable selinux.

   Open /etc/selinux/config and set the SELINUX parameter disabled. The configuration would look like this:

   # This file controls the state of SELinux on the system.
   # SELINUX= can take one of these three values:
   #     enforcing - SELinux security policy is enforced.
   #     permissive - SELinux prints warnings instead of enforcing.
   #     disabled - No SELinux policy is loaded.
   SELINUX=disabled
   # SELINUXTYPE= can take one of these three values:
   #     targeted - Targeted processes are protected,
   #     minimum - Modification of targeted policy. Only selected processes are protected.
   #     mls - Multi Level Security protection.
   SELINUXTYPE=targeted
   

2. Change the hostname, and set locale and timezone.

   # hostnamectl set-hostname rpi-cm4
   # localectl set-locale en_US.UTF-8
   # localectl set-keymap us
   # localectl set-x11-keymap "" "" "" "caps:ctrl_modifier"
   # timedatectl set-timezone Asia/Seoul
   

3. Install epel-release and update.

   # dnf install epel-release
   # dnf update
   # dnf install rsync
   

4. Add admin user account.

   # useradd -c "Admin Name" -G wheel -m -U $ADMIN_NAME
   

   Follow SSh Settings and Security (including 2FA) to restrict access. Also add ssh public keys to ~/.ssh/authorized_keys to both root and $ADMIN_NAME.

5. Create network interface config files.

   The network configuration is managed by NetworkManager.service. The configuration files are in /etc/sysconfig/network-scripts. We are going to use the Gigabit Ethernet PHY of the CM4 module which is based on Broadcom BCM54210PE (which is named eth0) to connect to the internal network, and Microchip's LAN7800 USB 3.0 to GbE (Gigabit Ethernet Bridge) (which is named enp1s0u3) to connect to the router.

   Create /etc/sysconfig/network-scripts/ifcfg-enp1s0u3:

   TYPE=Ethernet
   PROXY_METHOD=none
   BROWSER_ONLY=no
   BOOTPROTO=dhcp
   DEFROUTE=yes
   IPV4_FAILURE_FATAL=no
   IPV6INIT=no
   IPV6_DEFROUTE=yes
   IPV6_FAILURE_FATAL=no
   IPV6_ADDR_GEN_MODE=stable-privacy
   NAME=enp1s0u3
   DEVICE=enp1s0u3
   ONBOOT=yes
   

   Create /etc/sysconfig/network-scripts/ifcfg-eth0:

   TYPE=Ethernet
   PROXY_METHOD=none
   BROWSER_ONLY=no
   BOOTPROTO=static
   DEFROUTE=yes
   IPV4_FAILURE_FATAL=no
   IPV6INIT=no
   IPV6_DEFROUTE=yes
   IPV6_FAILURE_FATAL=no
   IPV6_ADDR_GEN_MODE=stable-privacy
   NAME=eth0
   DEVICE=eth0
   ONBOOT=yes
   IPADDR=192.168.100.254
   PREFIX=24
   

   Restart the NetworkManager service:

   # systemctl restart NetworkManager
   

6. Configure firewalld to allow NAT.

   Enable IP forwarding:

   # sysctl -w net.ipv4.ip_forward=1
   # echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/ip_forward.conf
   

   Add the public interface enp1s0u3 to the external zone:

   # firewall-cmd --change-interface=enp1s0u3 --zone=external
   # firewall-cmd --list-all --zone=external
   # firewall-cmd --runtime-to-permanent
   

   Add the internal interface eth0 to the trusted zone:

   # firewall-cmd --change-interface=eth0 --zone=trusted
   # firewall-cmd --list-all --zone=trusted
   # firewall-cmd --runtime-to-permanent
   

   All is done!

7. Install GUI, XRDP, and Firefox.

   Install Xfce:

   # dnf groupinstall Xfce
   # systemctl set-default graphical
   # reboot
   

   Install XRDP from EPEL:

   # dnf install xrdp
   # systemctl enable --now xrdp
   

   Install Firefox:

   # dnf install firefox
   

Fixing an Issue With Wifi Driver

The wifi driver that comes with AlmaLinux does not load.

$ journalctl -b | grep brcmfmac
Jun 22 20:11:51 rpi-cm4 kernel: brcmfmac: F1 signature read @0x18000000=0x15264345
Jun 22 20:11:51 rpi-cm4 kernel: brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
Jun 22 20:11:51 rpi-cm4 kernel: brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-compute-module.bin failed with error -2
Jun 22 20:11:51 rpi-cm4 kernel: usbcore: registered new interface driver brcmfmac
Jun 22 20:11:51 rpi-cm4 kernel: brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.txt failed with error -2
Jun 22 20:11:52 rpi-cm4 kernel: brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50

This post suggests that the default driver is corrupt. Following the answer in that post (copied here),

There's a comment here about this error being caused by corrupt driver files. I downloaded a fresh copy of the driver for brcmfmac43455 from here and everything worked fine.

The relevant driver(s) are under /lib/firmware/brcm:

$ ls -l brcmfmac43455*
-rw-r--r--. 1 root root 1723 May 11 00:44  brcmfmac43455-sdio.acepc-t8.txt
lrwxrwxrwx. 1 root root   31 May 11 00:44  brcmfmac43455-sdio.bin -> ../cypress/cyfmac43455-sdio.bin
lrwxrwxrwx. 1 root root   36 May 11 00:44  brcmfmac43455-sdio.clm_blob -> ../cypress/cyfmac43455-sdio.clm_blob
-rw-r--r--. 1 root root 2510 May 11 00:44 'brcmfmac43455-sdio.MINIX-NEO Z83-4.txt'
lrwxrwxrwx. 1 root root   49 May 11 00:44  brcmfmac43455-sdio.raspberrypi,3-model-a-plus.txt -> brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt
-rw-r--r--. 1 root root 1884 May 11 00:44  brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt
-rw-r--r--. 1 root root 1883 May 11 00:44  brcmfmac43455-sdio.raspberrypi,4-model-b.txt
lrwxrwxrwx. 1 root root   44 May 11 00:44 'brcmfmac43455-sdio.Raspberry Pi Foundation-Raspberry Pi 4 Model B.txt' -> brcmfmac43455-sdio.raspberrypi,4-model-b.txt
lrwxrwxrwx. 1 root root   44 May 11 00:44 'brcmfmac43455-sdio.Raspberry Pi Foundation-Raspberry Pi Compute Module 4.txt' -> brcmfmac43455-sdio.raspberrypi,4-model-b.txt

I downloaded brcmfmac43455-sdio.bin, brcmfmac43455-sdio.clm_blob, and brcmfmac43455-sdio.txt, and replaced/placed them under /lib/firmware/brcm. Now the command journalctl -b | grep brcmfmac prints

$ journalctl -b | grep brcmfmac
Jun 22 20:11:52 rpi4-router kernel: brcmfmac: F1 signature read @0x18000000=0x15264345
Jun 22 20:11:52 rpi4-router kernel: brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
Jun 22 20:11:52 rpi4-router kernel: brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-compute-module.bin failed with error -2
Jun 22 20:11:52 rpi4-router kernel: usbcore: registered new interface driver brcmfmac
Jun 22 20:11:52 rpi4-router kernel: brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
Jun 22 20:11:52 rpi4-router kernel: brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
Jun 22 20:11:52 rpi4-router kernel: brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Jan  4 2021 19:56:29 version 7.45.229 (617f1f5 CY) FWID 01-2dbd9d2e
Jun 22 20:11:57 rpi4-router NetworkManager[313]: <info>  [1529665917.6077] rfkill0: found Wi-Fi radio killswitch (at /sys/devices/platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/ieee80211/phy0/rfkill0) (driver brcmfmac)
Jun 22 20:11:58 rpi4-router kernel: brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
Jun 22 20:11:58 rpi4-router kernel: brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled

$ nmcli dev status
DEVICE         TYPE      STATE         CONNECTION
enp1s0u3       ethernet  connected     enp1s0u3
wlan0          wifi      disconnected  --
p2p-dev-wlan0  wifi-p2p  disconnected  --
eth0           ethernet  unavailable   --
lo             loopback  unmanaged     --

$ nmcli dev wifi list
IN-USE  BSSID              SSID         MODE   CHAN  RATE        SIGNAL  BARS  SECURITY
        DC:A4:CA:F1:09:04  --           Infra  1     195 Mbit/s  100     ▂▄▆█  WPA2
        DE:04:09:F1:CA:A0  KM_Guest     Infra  1     195 Mbit/s  100     ▂▄▆█  WPA1 WPA2
        88:3C:1C:C1:DC:65  --           Infra  8     130 Mbit/s  100     ▂▄▆█  WPA1 WPA2
        BA:3C:1C:C1:DC:64  --           Infra  149   270 Mbit/s  100     ▂▄▆█  WPA1 WPA2
        88:3C:1C:C1:DC:64  --           Infra  149   270 Mbit/s  100     ▂▄▆█  WPA1 WPA2
        00:27:1C:43:77:6A  SK_WiFi7768  Infra  3     270 Mbit/s  90      ▂▄▆█  WPA1 WPA2
        02:27:1C:43:77:6A  --           Infra  3     270 Mbit/s  90      ▂▄▆█  WPA2
        DE:05:09:F1:CA:A0  KM_Guest     Infra  149   405 Mbit/s  47      ▂▄__  WPA1 WPA2
        00:27:1C:36:3C:B7  SK_WiFi3CB5  Infra  4     270 Mbit/s  45      ▂▄__  WPA1 WPA2
        DC:A4:CA:F1:09:05  --           Infra  149   405 Mbit/s  44      ▂▄__  WPA2

Accessing Server's BMC Web Server

The SSH access to the Raspberry Pi module is through

$ ssh -p 47 $UNAME@hpc

And this is the only port open to this module. In addition, the external zone only accepts the ssh service.

To make things easier, we will add the node's BMC IP's to /etc/hosts

192.168.100.10      sensor
192.168.100.100     node0
192.168.100.101     node1
192.168.100.102     node2
192.168.100.103     node3
192.168.100.104     node4
192.168.100.105     node5
192.168.100.106     node6
192.168.100.107     node7
192.168.100.108     node8

There are two ways to access the BMC web server: (1) SSH local port forwarding and (2) XRDP.

SSH Local Port Forwarding

How to Set up SSH Tunneling (Port Forwarding) explains the local port forwarding very well. For example, to connect to node0's BMC web server,

$ ssh -p 47 -L 127.0.0.1:8080:node0:443 hpc

Open the Safari browser in private mode and type in https://127.0.0.1:8080. The reason to use the port number 8080 is because ports numbers less than 1024 are privileged ports and can be used only by root. That is, I would have done sudo ssh -p 47 -L 127.0.0.1:443:node0:443 hpc.

If I want to access other nodes, then I need to close the existing one and connect to a new one, or I can connect to new one using different port in the ssh client.

XRDP

We can connect to the GUI environment through the Microsoft Remote Desktop app. Again, the local port forwarding is required. The default XRDP port is 3389.

$ ssh -p 47 -L 127.0.0.1:3389:127.0.0.1:3389 hpc

In the Microsoft Remote Desktop app, connect to 127.0.0.1. Open the Firefox and point to the web server's address, e.g., https://node0.

---

Raspberry Pi Router

Set Hostname

```shell
# hostnamectl set-hostname rpi_router
```

SSH Tunneling

For ssh tunneling, install netcat

# dnf install netcat

To use the ssh tunneling through this router, edit .ssh/config in your local machine to include something like this:

Host awesome_name
IdentityFile ~/.ssh/id_rsa
HostName awesome_name.example.com
ProxyCommand ssh -o 'ForwardAgent yes' rpi_router 'nc %h %p'

DNS and DHCPD Services

dnsmasq seems to be a good choice. This will give us a good guideline.

Install dnsmasq

# dnf install dnsmasq

Edit /etc/dnsmasq.conf to enable:

# ...
conf-file=/usr/share/dnsmasq/trust-anchors.conf
dnssec
# ...
listen-address=192.168.101.254
# ...
dhcp-range=192.168.101.101,192.168.101.200,255.255.255.0,24h
# ...
dhcp-host=xx:xx:xx:xx:xx:xx,hostname_1,192.168.101.1,24h
dhcp-host=yy:yy:yy:yy:yy:yy,hostname_2,192.168.101.2,24h
dhcp-host=zz:zz:zz:zz:zz:zz,hostname_3,192.168.101.3,24h

Edit /etc/hosts to add:

192.168.101.1  hostname_1.example.com
192.168.101.2  hostname_2.example.com
192.168.101.3  hostname_3.example.com

The setting is so that it reads /ets/hosts to return ip addresses for the queried domains.

Check the configuration:

# dnsmasq --test

and enable and start the service

# systemctl enable --now dnsmasq.service

To see if dnsmasq started properly, check the system's journal:

# journalctl -u dnsmasq.service

Port Forwarding

See 5.9. Port Forwarding.

Redirect the port 47 to the ssh port 22 of a local machine:

# firewall-cmd --zone=external --add-forward-port=port=47:proto=tcp:toport=22:toaddr=192.168.101.2
# firewall-cmd --zone=external --add-masquerade

List forwarded ports:

# firewall-cmd --list-forward-ports --zone=external
# firewall-cmd --zone=external --list-all

Make the new settings persistent:

# firewall-cmd --runtime-to-permanent

To remove redirect:

# firewall-cmd --zone=external --remove-forward-port=port=47:proto=tcp:toport=22:toaddr=192.168.101.2
# firewall-cmd --zone=external --remove-masquerade

WiFi Hotspot

Edit /etc/sysconfig/network-scripts/ifcfg-Hotspot to include

ESSID=KM_Guest
MODE=Ap
KEY_MGMT=WPA-PSK
WPA_ALLOW_WPA2=yes
CIPHER_PAIRWISE=CCMP
CIPHER_GROUP=CCMP
MAC_ADDRESS_RANDOMIZATION=default
TYPE=Wireless
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=Hotspot
UUID=15dfc697-92a9-4d56-b037-028d74601d43
DEVICE=wlan0
ONBOOT=yes
IPADDR=10.10.0.1
PREFIX=24

Edit /etc/sysconfig/network-scripts/keys-Hotspot to include

WPA_PSK=strongPassword1234

Modify access permission

# chmod 0600 /etc/sysconfig/network-scripts/keys-Hotspot

Add the hotspot interface wlan0 to the nm-shared zone:

# firewall-cmd --change-interface=wlan0 --zone=nm-shared
# firewall-cmd --list-all --zone=nm-shared
# firewall-cmd --runtime-to-permanent

Remove the ssh service from the nm-shared zone:

# firewall-cmd --remove-service=ssh --zone=nm-shared
# firewall-cmd --list-all --zone=nm-shared
# firewall-cmd --runtime-to-permanent

Edit /etc/dnsmasq.conf to include

...
listen-address=...,10.10.0.1
...
dhcp-range=10.10.0.10,10.10.0.100,12h
...

---

BMC Network Setup

The server series is S2600.

• Management Guide for Intel Server Platforms with Embedded Integrated BMC
• Integrated Baseboard Management Controller Embedded Web Server
• Integrated Baseboard Management Controller (Integrated BMC) Web Console User Guide
• User Guide for Intel System Configuration Utility

Save and Restore System Configuration utility (SYSCFG) is available here.

In the head node, download it in a temporary directory and follow these steps

# unzip syscfg_v14_1_build32_allos.zip
# cd syscfg-v14.1_B32/Linux_x64/RHEL
# unzip syscfg.zip
# chmod +x syscfg
# mkdir -p /opt/syscfg/V14.1-B32/bin
# cp syscfg /opt/syscfg/V14.1-B32/bin
# cp -r ../../Licenses /opt/syscfg/V14.1-B32

The following applies to all nodes. We are going to open the LAN channel 1 (NIC1), which is connected to the cluster private network.

Set the name and password of UserID 2, enable it to LAN channel 1 and set the privilege

# /opt/syscfg/V14.1-B32/bin/syscfg /u 2 admin super-strong-password
# /opt/syscfg/V14.1-B32/bin/syscfg /ue 2 enable 1
# /opt/syscfg/V14.1-B32/bin/syscfg /up 2 1 admin sol

The "Complex Password" setting might be disabled in the BIOS.

Check if the access and privilege on LAN channel 1 are set

# /opt/syscfg/V14.1-B32/bin/syscfg /d user 2 1

which will show something like

User Configuration Settings:
----------------------------
User ID Selected: 2
Channel Selected: 1
User Name: admin
User Status: ENABLE
Privilege Level Limit: ADMIN
SOL Enable: ENABLE

To configure the network of LAN channel 1 (be sure to substitute ? with the node number)

# /opt/syscfg/V14.1-B32/bin/syscfg /le 1 static 192.168.100.10? 255.255.255.0 # Static IP
# /opt/syscfg/V14.1-B32/bin/syscfg /lc 1 12 192.168.100.254                   # Gateway
# /opt/syscfg/V14.1-B32/bin/syscfg /d lan 1                                   # Display settings

The node number starts from 0 (head node).

---

Yoctopuce's Environmental Sensors

This section describes setup for the temperature monitoring of the server room.

For temperature reading, we use yoctopuce's product Yocto-Meteo-V2 in conjunction with a network host YoctoHub-Ethernet that broadcasts the sensor readings and configurations over the web interface.

The list of parts ordered are

Reference	Description and Harmonized Tariff Code	Unit price (USD)	Quantity
METEOMK2	Yocto-Meteo-V2 (9032.9000 CH; Part of controlling apparatus)	57.01	1
BX1LBFTO	YoctoBox-Long-Thin-Black-Vents (3923.1000 FR; Plastic enclosure)	10.16	1
YHUBETH1	YoctoHub-Ethernet (9032.9000 CH; Part of controlling apparatus)	112.79	1
BXETHTSP	YoctoBox-HubEth-Transp (3923.1000 CH; Plastic enclosure)	18.29	1
USBA0050	USB-A-MicroB-50 (8544.4290 CN; USB cable)	3.05	1
YOTGBB20	USB-OTG-MicroB-MicroB-20 (8544.4290 CN; USB cable)	3.05	1

The manuals are at

• YoctoHub-Ethernet : User's guide
• Yocto-Meteo-V2 : User's guide

Software references are at

• VirtualHub
• Libraries

To connect to YoctoHub-Ethernet, type in the comment

$ ssh -Np 47 -L 127.0.0.1:8080:sensor:80 hpc

and open the browser and go to http://127.0.0.1:8080.