Remove support for cpuset MoM (pbs_mom.cpuset)

Follow the PBS Pro Design Document Guidelines.

Overview

The special pbs_mom.cpuset binary is only used in HPE systems MC990X/UV300 and ICE-X to run jobs using cpusets. There's special code in pbs_mom that is enclosed in #ifdef MOM_CPUSET that is compiled to create pbs_mom.cpuset. Now cgroups also supports cpusets, and running the standard pbs_mom with pbs_cgroups hook enabled works just as well.

Instead of running pbs_mom.cpuset, one can just enable the pbs_cgroups hook using the following steps:

  1. qmgr -c "export hook pbs_cgroups application/x-config default pbs_cgroups.json"
    This will create a file called "pbs_cgroups.json" in the current directory.
  2. Edit the pbs_cgroups.json file and set "vnode_per_numa_node" to true
    Example: "vnode_per_numa_node" : true
  3. Edit the pbs_cgroups.json file and set "use_hyperthreads" to true
    Example: "use_hyperthreads" : true
  4. If memory subsystem of cgroups is not mounted on the system, disable 'memory' and 'memsw' from the hook config file. That is,

# mount | grep mount

 ← returns empty, and not something like "cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory"

If the mount returns empty, then edit the pbs_cgroups.json file such that 'enabled' values for 'memory' and 'memsw' under cgroup are false:

"cgroup": {

  ...

"memory": {

"enabled": false,

...

"memsw": {

"enabled": false,

5. Re-import the updated hook config file:
    qmgr -c "import hook pbs_cgroups application/x-config default pbs_cgroups.json"

6. Enable the cgroups hook:
    qmgr -c "set hook pbs_cgroups enabled=true"

7. Restart pbs_mom so it can get the latest hook information:

   # ps -eaf | grep pbs_mom

  <mom_pid> ... /opt/pbs/sbin/pbs_mom

  # kill <mom_pid>

 # /opt/pbs/sbin/pbs_mom


For sites with large NUMA machines and machines with GPUs, it is recommended to set vnode_per_numa_node to true.




OSS Site Map

Project Documentation Main Page

Developer Guide Pages