Remove support for cpuset MoM (pbs_mom.cpuset)
- Lisa Nishiyama
- Albeaus R Bayucan
Follow the PBS Pro Design Document Guidelines.
Links
- Link to discussion on Developer Forum: http://community.pbspro.org/t/remove-support-for-cpuset-mom/2066
- Link to a related design about replacing a cpuset optimization: To optimize job resource polling, discontinue reporting of resources_used values of PBS root jobs
- Link to pull request: <PR link if available>
Overview
The special pbs_mom.cpuset binary is only used in HPE systems MC990X/UV300 and ICE-X to run jobs using cpusets. There's special code in pbs_mom that is enclosed in #ifdef MOM_CPUSET that is compiled to create pbs_mom.cpuset. Now cgroups also supports cpusets, and running the standard pbs_mom with pbs_cgroups hook enabled works just as well.
Instead of running pbs_mom.cpuset, one can just enable the pbs_cgroups hook using the following steps:
- qmgr -c "export hook pbs_cgroups application/x-config default pbs_cgroups.json"
This will create a file called "pbs_cgroups.json" in the current directory. - Edit the pbs_cgroups.json file and set "vnode_per_numa_node" to true
Example: "vnode_per_numa_node" : true - Edit the pbs_cgroups.json file and set "use_hyperthreads" to true
Example: "use_hyperthreads" : true - If memory subsystem of cgroups is not mounted on the system, disable 'memory' and 'memsw' from the hook config file. That is,
# mount | grep mount
← returns empty, and not something like "cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory"
If the mount returns empty, then edit the pbs_cgroups.json file such that 'enabled' values for 'memory' and 'memsw' under cgroup are false:
"cgroup": {
...
"memory": {
"enabled": false,
...
"memsw": {
"enabled": false,
5. Re-import the updated hook config file:
qmgr -c "import hook pbs_cgroups application/x-config default pbs_cgroups.json"
6. Enable the cgroups hook:
qmgr -c "set hook pbs_cgroups enabled=true"
7. Restart pbs_mom so it can get the latest hook information:
# ps -eaf | grep pbs_mom
<mom_pid> ... /opt/pbs/sbin/pbs_mom
# kill <mom_pid>
# /opt/pbs/sbin/pbs_mom
For sites with large NUMA machines and machines with GPUs, it is recommended to set vnode_per_numa_node to true.
Project Documentation Main Page