Reconfirming degraded reservations that are running
- Bhroam A Mann
- Arun Grover (Deactivated)
Follow the PBS Pro Design Document Guidelines.
Links
- Link to discussion on Developer Forum: http://community.pbspro.org/t/reconfirmation-of-running-reservations/1872
Overview
Currently in PBS, degraded reservations can only be reconfirmed before they start. If a reservation starts degraded when it starts running, it will be degraded through the lifetime of that occurrence. PBS will be enhanced to reconfirm reservations that are running. This applies to degraded reservation and reservations in-conflict.
Technical Details
If a non-running degraded reservation is reconfirmed, its node solution can completely change. When a reservation is not running, where it is going to run does not matter. We only care that the nodes given to the reservation satisfy the select spec. Reconfirming a running reservation is different. There might be jobs running on nodes, so only the unavailable nodes can be replaced. All other nodes must remain with the reservation.
Changes to how degraded reservations are reconfirmed
There are the following attributes which affect how reservations are reconfirmed:
reserve_retry_init (server) - Amount of time after nodes become unavailable before the first reconfirmation attempt is made (default 2hr).
reserve_retry_cutoff (server) - Amount of time before a reservation starts where we stop trying to reconfirm (default 2hr).
resv_retry (reservation) - Epoch time of the next time reconfirmation attempt will be made.
Current workflow of a degraded or in-conflict reservation
- When nodes of a reservation become unavailable, the server will wait resv_retry_init before attempting to reconfirm a reservation.
- After each unsuccessful reconfirmation attempt, we try again in half the time between then and the start of the reservation (e.g. if the reservation starts in one day, we wait half a day).
- We stop attempting to reconfirm the reservation at the start time - reserve_retry_cutoff
New workflow of a degraded or in-conflict reservation
The times between reconfirmation attempts will need to change. Since we are now reconfirming running reservations, setting the time between attempts to half the time to the start will no longer work.
- resv_retry_cutoff will no longer be used
- When a node becomes unavailable, PBS will wait reserve_retry_time seconds before trying the first reconfirmation attempt
- After an unsuccessful reconfirmation attempt is made, PBS will reset resv_retry to the next reconfirmation time.
- One reconfirmation attempt will be made right before the reservation starts.
- Once a reservation starts, we will attempt to reconfirm it every reserve_retry_time seconds.
- If a node of a running reservation goes into state offline or maintenance and there is a job running in the reservation on that node, the reservation will not be reconfirmed until those jobs are finished.
- Running reservations in state in-conflict will not be reconfirmed.
- Standing in-conflict reservations can be reconfirmed once an occurrence ends and before the next occurrence starts.
Changes to the external interface
Server attribute reserve_retry_time
Perms: Manager write / Everyone read
Status: New - An attempt to reconfirm a degraded reservation every reserve_retry_time seconds. The default is 600s.
Server attribute reserve_retry_init
Status: Deprecated - The first attempt to reconfirm a degraded reservation resv_retry_time seconds after it was originally degraded.
Server attribute reserve_retry_cutoff
Status: obsolete
Reservation attribute: reserve_retry
Perms: Manager read
Description: Epoch time of the next reconfirmation attempt
Accounting log: 'Y' record
Description: Every time a degraded reservation is reconfirmed (running or not running) PBS server will log a 'Y' record in the accounting logs.
This 'Y' record will have the following format - Y;<resvID>;requestor=Scheduler@<server> start=<(new/original) start time> end=<(new/original) end time> nodes=(<allotted nodes>)
Project Documentation Main Page