PP-325: Support Cgroups

Note: PP-810 contains additional updates to the cgroup design.

Overview:

The objective of  PP-325 - Getting issue details... STATUS is to provide a hook that restricts resource availability for job processes by utilizing the Linux cgroup infrastructure. An introductory video to cgroups together with links to other resources may be found here: https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups The definitive Linux kernel cgroup documentation may be found here: https://www.kernel.org/doc/Documentation/cgroup-v1/ It is assumed that the reader is familiar with the various cgroup subsystems and how they may be utilized. When the cgroup hook is enabled, it runs on every node assigned to the job that supports cgroups. When a job is started, the hook creates a set of directories for the configured subsystems based on the resource requirements of the job and then places the job process within the cgroup. The kernel then enforces resource restrictions based on the newly created cgroup settings while the job is running. The cgroup hook may be configured to periodically poll the job's cgroup while the job is running and updates resource utilization. When the job completes, the final resource utilization measurement is taken and the hook removes the cgroup directories it created when the job was initialized.

Interface 1: cgroup configuration file

• Visibility: Public
• Change Control: Experimental

• Synopsis: JSON hook config file

• Details: The configuration file allows the administrator to adjust the behavior of the cgroup hook across their cluster. The file must conform to JSON syntax. A sample configuration file is displayed and described below:

{
    "cgroup_prefix"         : "pbspro",
    "exclude_hosts"         : ["node001", "node002"],
    "exclude_vntypes"       : ["disable_cgroups", "login_node"],
    "run_only_on_hosts"     : [],
    "periodic_resc_update"  : true,
    "vnode_per_numa_node"   : false,
    "online_offlined_nodes" : true,
    "use_hyperthreads"      : false,
    "ncpus_are_cores"       : false,
    "cgroup" : {
        "cpuacct" : {
            "enabled"         : true,
            "exclude_hosts"   : ["node003"],
            "exclude_vntypes" : ["red_node"]
        },
        "cpuset" : {
            "enabled"         : true,
            "exclude_hosts"   : ["node004"],
            "exclude_vntypes" : ["green_node"]
        },
        "devices" : {
            "enabled"         : false,
            "exclude_hosts"   : [],
            "exclude_vntypes" : [],
            "allow"           : [
                "b *:* rwm",
                "c *:* rwm",
                ["mic/scif", "rwm"],
                ["nvidiactl", "rwm", "*"],
                ["nvidia-uvm", "rwm"]
            ]
        },
        "hugetlb" : {
            "enabled"         : false,
            "exclude_hosts"   : [],
            "exclude_vntypes" : [],
            "default"         : "0MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "0MB"
        },
        "memory" : {
            "enabled"         : true,
            "exclude_hosts"   : [],
            "exclude_vntypes" : ["blue_node"],
            "soft_limit"      : false,
            "default"         : "256MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "1GB"
        },
        "memsw" : {
            "enabled"         : true,
            "exclude_hosts"   : [],
            "exclude_vntypes" : ["grey_node"],
            "default"         : "256MB",
            "reserve_percent" : "0",
            "reserve_amount"  : "1GB"
        }
    }
}

Global Parameters:

The global parameters are used to modify the behavior of the cgroup hook across all nodes in the PBS Pro complex. They are not specific to any cgroup subsystem.

cpuacct Subsystem:

The cpuacct subsystem provides support for measuring CPU usage within a cgroup.

cpuset Subsystem:

The cpuset subsystem is used by the Linux kernel to restrict access to both CPU and memory based on available sockets and NUMA nodes.

devices Subsystem:

The devices subsystem is used to grant or restrict access to devices on the system. This is most commonly used for accelerator cards such as GPUs, and MICs.

hugetlb Subsystem:

The hugetlb subsystem restricts the amount of huge page memory that may be used in a cgroup.

memory Subsystem:

The memory subsystem restricts the amount of physical memory all of the processes in a cgroup may allocate. It works in conjunction with the memsw (memory and swap) subsystem to limit both physical memory and virtual memory. The memsw subsystem should be enabled if the memory subsystem is enabled. Otherwise, the virtual memory limit will remain unrestricted. If the processes in a cgroup exceed their physical memory limit, the kernel will begin to utilize swap space even if the node has sufficient physical memory to allocate. When a job specifies a mem limit without a vmem limit, the vmem limit is automatically set to the mem limit.

memsw Subsystem:

The memsw subsystem restricts the amount of virtual memory all of the processes in a cgroup may allocate. This includes both physical memory and swap space. The vmem resource may be used to specify the virtual memory limit. The value should always be greater than or equal to the amount of mem requested. The vmem resource must be added to the resources line in the sched_config file in order to schedule jobs based on virtual memory.

Interface 2: nmics

• Visibility: Public
• Change Control: Experimental

• Synopsis: Integer resource used to request the number of Intel MIC devices required per chunk

• Details: Users may submit jobs requesting nmics=<#> in order for the scheduler to assign nodes with a sufficient number of Intel MIC devices. The nmics resource must be added to the resources line in the scheduler configuration file for this resource to be considered for scheduling.
  

Interface 3: ngpus

• Visibility: Public
• Change Control: Experimental

• Synopsis: Integer resource used to request the number of NVIDIA GPUs required per chunk

• Details: Users may submit jobs requesting ngpus=<#> in order for the scheduler to assign nodes with a sufficient number of NVIDIA GPU devices. The ngpus resource must be added to the resources line in the scheduler configuration file for this resource to be considered for scheduling.
  

Interface 4: vmem

• Visibility: Public
• Change Control: Experimental

• Synopsis: Size resource used to request the amount of virtual memory required per chunk

• Details: Users may submit jobs requesting vmem=<size> in order for the scheduler to assign nodes with a sufficient amount of virtual memory. The vmem resource must be added to the resources line in the scheduler configuration file for this resource to be considered for scheduling. MoM must be restarted when the memsw subsystem is enabled before it will report a value for vmem.

Interface 5: hpmem

• Visibility: Public
• Change Control: Experimental

• Synopsis: Size resource used to request the amount of huge page memory required per chunk

• Details: Users may submit jobs requesting hpmem=<size> in order for the scheduler to assign nodes with a sufficient amount of huge page memory. The hpmem resource must be added to the resources line in the scheduler configuration file for this resource to be considered for scheduling. MoM must be restarted when the hugetlb subsystem is enabled before it will report a value for hpmem.

Administrator Notes

The cpuset MoM that is available for SGI systems utilizes libcpuset to manage cpusets for jobs. The cgroup hook provides similar functionality when the cpuset subsystem is enabled. Administrators should avoid using the cpuset MoM in conjuction with the cgroup hook.