Overview
Placement sets are used to force a job to run on all nodes that are alike in certain ways. Placement sets are created based on resources of type string_array. The resource is set on nodes. A placement set is a set of nodes where the resource has the same value.
Placement sets can be enabled at the server, queue, or job level. Up until now the whole job must fit within a single placement set. This feature will allow each chunk to request placement setsThere is a desire for jobs to be placed on nodes of a single type. The type groupings can have many values. Job submitters do not care which value of the group their job is placed on as long as all the nodes for the job have the same value for the group. In PBS today, this grouping is called a placement set. Placement sets can be defined complex wide, queue wide, or per-job. The more specific placement sets have priority the less specific (e.g. queue placement sets have priority over the complex-wide sets). The new desire is more finely grained grouping. Instead of per-job, it should be per-rank or set of ranks.
Technical Details
Glossary
...
Currently a job can be run within one pool of placement sets. These will come from the server, queue, or job. The job's place=group overrides the queue's node_group_key which overrides the server's node_group_key. Per-chunk placement sets will work slightly differentlysimilarly. If a chunk complex requests a group, it will override any other placement sets for the job. If both per-job and per-chunk placement are requested together, per-chunk placement will be made by breaking the per-job placement sets aparta job has multiple chunk complexes where some request a group and others do not, the chunk complexes that did not request a group will be placed by the for the job.
Example:
| Nodes | Color | Shape |
|---|---|---|
| 1-2 | blue | square |
| 3-4 | blue | triangle |
| 5-6 | red | square |
| 7-8 | red | triangle |
...
If the server has node_group_key=color and a job requests select=group=shape, the job will be placed on:
color=blue+shape=square : nodes 1-2color=blue+, 5-6
shape=triangle : nodes 3-4
color=red+shape=square: nodes 5-6
color=red+shape=triangle: nodes 7-87-7
Example 2: Interaction between server's node_group_key with multiple chunk complexes. One chunk complex has per-chunk placement and the other does not.
...
Both chunk complexes will be placed similarly: color=blue (nodes 1-4) or color=red (nodes 5-8). The difference between per-job place=group=color and this request is that the two chunk complexes can be placed on different placement sets. It is possible for the first chunk complex to be placed on color=blue and the second chunk complex be placed on color=red
Interaction with
...
placement set spanning
Currently if no placement set is large enough (when empty) to fit a job, the job will span across all nodes. This can be controlled with the scheduler's do_not_span_psets attribute. If do_not_span_psets is true, and a job can not fit within any placement set, the job will never run.
...
If both per-job placement and per-chunk placement are requested, spanning will happen in a tiered fashion. We first try and place chunks using their per-job+ per-chunk placement sets. If any chunk can not fit, we will attempt to place the whole job in the per-job placement sets. If the job still can not fit, we will span as we do today over all nodes.
...