...
There is a desire for jobs to be placed on nodes of a single type. The type groupings can have many values. Job submitters do not care which value of the group their job is placed on as long as all the nodes for the job have the same value for the group. In PBS today, this grouping is called a placement set. Placement sets can be defined complex wide, queue wide, or per-just for the job. The more specific placement sets have priority overrides the less specific (e.g. queue placement sets have priority over override the complex-wide sets). The new desire is more finely grained grouping. Instead of per-job, it should be per-rank or set of ranks.
...
Currently a job can be run within one pool of placement sets. These will come from the server, queue, or job. The job's place=group overrides the queue's node_group_key which overrides the server's node_group_key. Per-chunk placement sets will work similarly. If a chunk complex requests a group, it will override any other placement sets for the job. If a job has multiple chunk complexes where some request a group and others do not, the chunk complexes that did not request a group will be placed by the for the per-job placement sets.
Example:
| Nodes | Color | Shape |
|---|---|---|
| 1-2 | blue | square |
| 3-4 | blue | triangle |
| 5-6 | red | square |
| 7-8 | red | triangle |
...
If no per-chunk placement is requested, per-job placement set spanning will work like it does today
If only per-chunk placement is requested, it will work similarly as per-job placement set spanning. It will only affect the chunk complex though. If the chunk complex can not fit into any placement set, it will span over all nodes. If multiple chunk complexes have different groups, spanning for each chunk complex is evaluated individually.
If both per-job placement and per-chunk placement are requested, spanning will happen in a tiered fashion. We first try and place chunks using their per-chunk placement sets. If any chunk can not fit, we will attempt to place the whole job in the per-job placement sets. If the job still can not fit, we will span as we do today over all nodesSpanning will not change. The decision to span will be made at the job level. Regardless if the entire job can't fit in any per-job placement set or a chunk can't fit in any per-chunk placement set, the entire job will span over all nodes. The interaction with do_not_span_psets will not change either.