PP-927: Update the installation procedure for PBS on Cray CLE 6.0 to use the PBS rpm

Forum Discussion

PP-927 - Getting issue details... STATUS

Overview:
This enhancement and the instructions below only apply to installing PBS Professional on the Cray CLE 6.0 system.

Behavior change:
During install, check to see if the system is a CLE 6.0, and if it is, then pbs_postinstall will not be called.  The externally visible behavior change on Cray (CLE 6.0 and newer) will be that some PBS directories and files will not be created or configured (e.g. PBS_HOME and /etc/pbs.conf).  Instead, for CLE 6.0 (and newer) the directories and files will be created when PBS is started for the first time.


Installation instructions for PBS Professional on CLE 6.0:
These instructions rely on Cray's Image Management and Provisioning System (IMPS) features.  These instructions are based on the information found in the section that covers "Install Third-Party Software with a Custom Image" of the Cray document S-2559 (https://pubs.cray.com/content/S-2559/CLE%206.0.UP04/xctm-series-software-installation-and-configuration-guide-cle-60up04-s-2559-rev-b/install-third-party-software-with-a-custom-image-recipe there is also a PDF available).  Other Cray documents that may be of use are listed below as references.  Please be aware that newer versions of the documents may exist.

Instruction to Install PBS Professional on CLE 6.0 with a custom image:

  1. Follow Cray's instructions for creating a repository (Refer to the section on "Install Third-Party Software with a Custom Image" in the Software Installation and Configuration Guide in Cray document S-2559)
  2. Follow Cray's instructions for creating a package collection in Cray document S-2559
    1. Use the pbspro_server-* RPM.
  3. Follow Cray's instructions (in Cray document S-2559) for creating a new recipe and updating it with the PBS Professional RPM.  
    Cray requires that each recipe be based on the type of node where the image will run.  For example, a service node would be contain the service image (e.g. service_cle_6.0up01_sles_12_x86-64_ari) as a base sub-recipe.  Therefore, if PBS Professional server/scheduler/comm will be run on the sdb and the PBS Professional MoM on the login nodes, at least 2 different recipes will be needed.  One recipe for the server/sched/comm and one recipe for the MoMs.
    1. In addition to adding the PBS Professional RPM, add the other RPMs that PBS Professional requires:
      1. In order to satisfy /bin/sh, also add sle-server_<version>_x86-64
        smw# recipe update -r sle-server_12sp3_x86-64 <recipe name>
      2. As Cray's instructions state, the base and base_updates repository pairs should be added.  This means sle-server_<version>_x86-64_updates also needs to be added.
        smw# recipe update -r sle-server_12sp3_x86-64_updates <recipe name>
    2. Follow Cray's instructions (in Cray document S-2559) to validate the image recipe
    3. repeat as needed for the different types of nodes.
  4. Follow Cray's instructions (in Cray document S-2559) to build and push/package the image.
  5. Create any desired configuration sets to make /etc/pbs.conf persistent.  Please refer to Cray’s documentation on configuration sets and the configurator for more information.
    The Cray ansible play that would need editing is "simple sync" for /etc/pbs.conf.
    WARNING: Note that unlike in prior Cray CLE versions (CLE 5.2 and earlier), with CLE 6.0 /var/spool is not persistent by default. So therefore, when nodes are rebooted, any configuration changes that were made will be gone unless Cray’s cfgset command is used to make particular directories (for example /var/spool/pbs, /etc/pbs.conf, et cetera) persistent.   
    1. It may be necessary to create different persistent /etc/pbs.conf files depending on how the system will be configured.
      When making the /etc/pbs.conf file persistent be sure to include the following (at a minimum):
      PBS_EXEC=/opt/pbs
      PBS_SERVER=<hostname of the machine hosting the server>
      PBS_START_SERVER=<set this to 1 if server is to run on this host, 0 otherwise>
      PBS_START_SCHED=<set this to 1 if scheduler is to run on this host, 0 otherwise>
      PBS_START_COMM=<set this to 1 if a communication daemon is to run on this host, 0 otherwise>
      PBS_START_MOM=<set this to 1 if mom is to run on this host, 0 otherwise>
      PBS_HOME=/var/spool/pbs
      PBS_CORE_LIMIT=unlimited
      PBS_SCP=/usr/bin/scp
    2. Add any other configuration changes to /etc/pbs.conf that need to persistent across reboots
  6. Follow Cray's instructions (in Cray document S-2559) to assign the new boot image to the nodes where PBS Professional should be installed.  This includes all nodes running PBS Professional services (e.g. pbs_server, pbs_mom, etc.) and those requiring access to PBS Pro commands (e.g. qsub, qstat, etc.).  Include the configuration set (from step 5) that contains the information that needs to be persistent.
    smw# cnode update -i  /var/opt/cray/imps/boot_images/<image name>.cpio -c <configuration set name> <cname-of-node-to-update or group-to-update>
  7. Follow Cray's instructions to reboot the system (in Cray document S-2559)
  8. Log on to the nodes that are hosting PBS Professional daemons
    1. Start PBS Professional on that node:
      For example, on the server node:
      # PBS_DATA_SERVICE_USER=<non-root user for the database> /etc/init.d/pbs start
      Reminder: PBS will only report compute nodes that are up and in batch mode.
    2. Stop PBS Professional so the PBS_HOME directory can be made persistent
  9. Make PBS_HOME (e.g. /var/spool/pbs) persistent.  
    Please refer to Cray’s documentation on configuration sets and the configurator for more information.  The Cray ansible play that would need editing are "Persistent Dirs".  Modify the same configuration set that was created earlier in step 5.
    WARNING: Note that unlike in prior Cray CLE versions (CLE 5.2 and earlier), with CLE 6.0 /var/spool is not persistent by default. So therefore, when nodes are rebooted, any configuration changes that were made will be gone unless Cray’s cfgset command is used to make particular directories (for example /var/spool/pbs, /etc/pbs.conf, et cetera) persistent.   
  10. Start PBS Professional.  At this point PBS Professional will be up and running, connected to ALPS and ready to be configured (for example, add moms to the server, set licenses, etc.). 

    Note: A PBS Professional module file is located in PBS_EXEC/etc/modulefile.  For systems utilizing modules, administrators may choose to manually copy this file to an appropriate location based on their system configuration.


Cray documentation references:

XC Series Software Installation and Configuration Guide S-2559

XC Series Configurator User Guide S-2560

XC Series DVS Administration Guide S-0005


Site Map

Developer Guide Pages