...
Currently a job can be run within one pool of placement sets. These will come from the server, queue, or job. The job's place=group overrides the queue's node_group_key which overrides the server's node_group_key. Per-chunk grouping will work similarly. If a chunk complex requests a group, it will override any other placement sets for the job. If a job has multiple chunk complexes where some request a group and others do not, the chunk complexes that do not request a group will be placed over all nodes available to the job (e.g. if the job is in a queue with nodes associated with it, only those nodes). It is invalid to to request place=group and per-chunk grouping.
Example:
| Nodes | Color | Shape |
|---|---|---|
| 1-2 | blue | square |
| 3-4 | blue | triangle |
| 5-6 | red | square |
| 7-8 | red | triangle |
Current:
If the server has node_group_key=color and a job requests place=group=shape, the job will be placed
...
If the job requests per-chunk grouping, and any chunk can not fit, the entire job will span. This is regardless if other chunks can fit in their placement sets.
Interaction with only_explicit_psets
The only_explicit_psets scheduler attribute tune the placement set creation. If any node does not have a grouping resource set on it, it is usually added to a resource="" (e.g. color="") placement set. If only_explicit_psets is true, then the resource="" placement set is not created, and those nodes are not available for placement.
There is no change in this behavior with per-chunk grouping. If only_explicit_psets is true, then the resource="" placement set is not created for that chunk complex's placement pool.
Clarifications
The nodes used to create placement sets:
- If the job is in a queue with nodes associated with it (node's queue attribute), only the nodes associated with the queue are available
- If the job is not in a queue with nodes associated with it, but there are other queues with nodes associated with them, then all nodes that are not associated with queues are available
- If there are no nodes associated with any queue, then all nodes are available.
Restrictions
- Per-chunk grouping is incompatible with place=pack. With place=pack, all chunks must be placed on one host.
- Per-chunk grouping is incompatible with place=group. You can either have per-job placement or per-chunk placement, but not both.