Technical Description

Compute Node Boot Procedure

The Scyld cluster architecture is designed around light-weight provisioning of compute nodes using the master node's kernel and Linux distribution. Network booting ensures that what is provisioned to each compute node is properly version-synchronized across the cluster.

Earlier Scyld distributions supported a 2-phase boot sequence. Following PXE boot of a node, a fixed Phase 1 kernel and initial RAM disk (initrd) were copied to the node and installed. Alternatively, this Phase 1 kernel and initrd were used to boot from local hard disk or removable media. This Phase 1 boot package then built the node root file system rootfs in RAM disk, requested the run-time (Phase 2) kernel and used 2-Kernel-Monte to switch to it, then loaded the Scyld daemons and initialized the BProc system. Means were provided for installing the Phase 1 boot package on local hard disk and on removable floppy and CD media.

Beginning with Scyld 30-series, PXE is the supported method for booting nodes into the cluster. For some years, all servers produced have supported PXE booting. For servers that cannot support PXE booting, Scyld ClusterWare provides the means to easily produce Etherboot media on floppy disk or CD to use as compute node boot media.

Scyld Beowulf Series 30 and Scyld ClusterWare HPC do not support booting of compute nodes from local hard disk, although this can be configured. If this is required, contact Scyld Customer Support.

The Boot Package

The compute node boot package consists of the kernel, initrd, and rootfs for each compute node. The beoboot command builds this boot package.

By default, the kernel is the one currently running on the master node. However, other kernels may be specified to the beoboot command and recorded on a node-by-node basis in the Beowulf configuration file. This file also includes the kernel command line parameters associated with the boot package. This allows each compute node to potentially have a unique kernel, initrd, rootfs, and kernel command lines.

Note that if you specify a different kernel to boot specific compute nodes, these nodes cannot be part of the BProc unified process space.

The path to the initrd and rootfs are passed to the compute node on the kernel command line, where it is accessible to the booting software.

Each time the Beowulf services restart on the master node, the beoboot command is run to recreate the default compute node boot package. This ensures that the package contains the same versions of the components as are running on the master node.

BProc Distributed Process Space

Scyld BeoMaster is able to provide a single system image through its use of BProc, the Scyld process space management kernel enhancement. BProc enables the processes running on cluster compute nodes to be visible and manageable on the master node. Processes start on the master node and are migrated to the appropriate compute node by BProc process migration code. Process parent-child relationships and UNIX job control information are maintained with the migrated jobs, as follows:

• All processes appear in the master node's process table.

• All standard UNIX signals (kill, suspend, resume, etc.) can be sent to any process on a compute node from the master.

• The stdin, stdout and stderr output from jobs is redirected back to the master through a socket.

BProc is one of the primary features that makes a Scyld cluster different from a traditional Beowulf cluster. It is the key software component that makes compute nodes appear as attached computational resources to the master node. The figure below depicts the role BProc plays in a Scyld cluster.

BProc itself is divided into two components:

• BPmaster — a daemon program that runs on the master node at all times

• BPslave — a daemon program that runs on each of the compute nodes

The BPmaster daemon uses a process migration module (VMADump in older Scyld systems or TaskPacker in newer Scyld systems) to freeze a running process so that it can be transferred to a remote node. The same module is also used by the BPslave daemon to thaw the process after it has been received. In a nutshell, the process migration module saves or restores a process's memory space to or from a stream. In the case of BProc, the stream is a TCP socket connected to the remote machine.

VMADump and TaskPacker implement an optimization that greatly reduces the size of the memory space required for storing a frozen process. Most programs on the system are dynamically linked; at run-time, they will use mmap to map copies of various libraries into their memory spaces. Since these libraries are demand paged, the entire library is always mapped even if most of it will never be used. These regions must be included when copying a process's memory space and included again when the process is restored. This is expensive, since the C library dwarfs most programs in size.

For example, the following is the memory space for the program sleep. This is taken directly from /proc/pid/maps.

08048000-08049000 r-xp 00000000 03:01 288816     /bin/sleep
08049000-0804a000 rw-p 00000000 03:01 288816     /bin/sleep
40000000-40012000 r-xp 00000000 03:01 911381     /lib/ld-2.1.2.so
40012000-40013000 rw-p 00012000 03:01 911381     /lib/ld-2.1.2.so
40017000-40102000 r-xp 00000000 03:01 911434     /lib/libc-2.1.2.so
40102000-40106000 rw-p 000ea000 03:01 911434     /lib/libc-2.1.2.so
40106000-4010a000 rw-p 00000000 00:00 0
bfffe000-c0000000 rwxp fffff000 00:00 0

The total size of the memory space for this trivial program is 1,089,536 bytes; all but 32K of that comes from shared libraries. VMADump and TaskPacker take advantage of this; instead of storing the data contained in each of these regions, they store a reference to the regions. When the image is restored, mmap will map the appropriate files to the same memory locations.

In order for this optimization to work, VMADump and TaskPacker must know which files to expect in the location where they are restored. The bplib utility is used to manage a list of files presumed to be present on remote systems.

Compute Node Categories

Each compute node in the cluster is classified into one of three categories by the master node: "configured", "ignored", or "unknown". The classification of a node is dictated by whether or where it is listed in one of the following files:

• The cluster config file /etc/beowulf/config (includes both "configured" and "ignored nodes")

• The unknown addresses file /var/beowulf/unknown_addresses (includes "unknown" nodes only)

When a compute node completes its initial boot process, it begins to send out DHCP requests on all the network interface devices that it finds. When the master node receives a DHCP request from a new node, the new node will automatically be added to the cluster as "configured" until the maximum configured node count is reached. After that, new nodes will be classified as "ignored". Nodes will be considered "unknown" only if BeoSetup isn't configured to auto-insert or auto-append new nodes.

The cluster administrator can change the default node classification behavior using the BeoSetup program or by manually editing the /etc/beowulf/config file. The classification of any specific node can also be changed manually by the cluster administrator. See the Chapter called Configuring the Cluster with BeoSetup for information on configuring the cluster with BeoSetup and the Chapter called Configuring the Cluster Manually for information on configuring the cluster manually. Also see the Appendix called Special Directories, Configuration Files, and Scripts to learn about special directories, configuration files, and scripts.

Following are definitions of the node categories.

Compute Node States

Cluster compute nodes may be in any of several functional states, such as down, up, or unavailable. Some of these states are transitional (boot or shutdown); some are informational variants of the up state (unavailable and error). BProc actually handles only 3 node operational variations:

• The node is not communicating — down. Variations of the down state may record the reason, such as halted (known to be halted by the master) or reboot (the master shut down the node with a reboot command).

• The node is communicating — up, unavailable, or error. Here the strings indicate different levels of usability.

• The node is transitioning — boot. This state has varying levels of communication, operating on scripted sequence.

During a normal power-on sequence, the user will see the node state change from down to boot to up. Depending on the machine speed, the boot phase may be very short and may not be visible due to the update rate of BeoStatus. All state information is reset to down whenever the BPmaster daemon is started/restarted.

In the following diagram, note that these states can also be reached via imperative commands such as bpctl. This command can be used to put the node into the error state, such as in response to an error condition detected by a script.

Following are definitions of the compute node states:

Miscellaneous Components

Scyld ClusterWare includes several miscellaneous components, such as name lookup functionality (BeoNSS), IP communications ports, library caching, and external data access.