Links

Link to discussion on Developer Forum: https://community.openpbs.org/t/design-document-for-reservation-tracking-hook-events/2649?u=pmrich
Link to development branch: https://github.com/argonne-lcf/openpbs/tree/reservation_begin_hook_event
Link to pull request: https://github.com/openpbs/openpbs/pull/2440

Overview

While there is an event for reservation ends (RESV_END) there is not one for when a reservation starts or is confirmed. And while there is a hook that can modify reservations at submission time, there isn't one that logs or can make modifications to reservation alteration requests. For tying into real-time accounting systems as well as tracking and reporting the progress of ongoing reservations, this set of changes adds the events RESV_CONFIRM, RESV_BEGIN, and MODIFYRESV that allows the site to easily enhance logging and tracking about advance reservations. The last hook, MODIFYRESV also allows hooks to run against pbs_ralter invocations and apply changes like one would for the RESVSUB hook, in a similar fashion to the relationship between the hooks run for qsub and qalter invocations. All three of these are being completed together due to how closely related these hooks are. A reverse lookup dictionary has also been added to ease translation of reservation states for hook writers.

Technical Details

Information for Hook Writers

RESV_BEGIN

resv_begin hooks are read-only
The type for the resv_begin event is pbs.RESV_BEGIN
RESV_BEGIN hooks run at the server
Hooks registered to the resv_begin event will be run after the queue has been started and enabled but before email notifications and the 'B' accounting record are written.
pbs.event().accept() and pbs.event().reject() both allow the reservation to proceed. This hook is read-only.
pbs.event().reject() should allow for a message to be sent to standard logging mechanisms.
Fields in the pbs.event().resv() object are the same as for the "resv_end" event.
A rejection in the resv_begin hook will not affect the start of the reservation at this time, as the scheduler will treat the reservation as begun independent of the reservation's status in the server, provided that the reservation is otherwise runnable.

RESV_CONFIRM

resv_confirm hooks are read-only.
The type for the resv_confirm event is pbs.RESV_CONFIRM.
resv_confirm hooks run at the server.
Hooks registered to the resv_confirm event will be run immediately before the "reservation confirmed" accounting record is written in req_confirmresv().
pbs.event().accept() and pbs.event().reject() both allow the reservation to proceed. This hook is read-only.
pbs.event().reject() should allow for a message to be sent to standard logging mechanisms.
Fields in the pbs.event().resv() object are the same as for the "resv_end" event.
A rejection in the resv_confirm hook will not affect reservation confirmation at this time.
resv_confirm hooks will be run on reservation re-confirmation.

MODIFYRESV

Mofidyresv hooks are read-write.
The type for the modifyresv event is pbs.MODIFYRESV.
The modifyresv event is run at the server.
Modifyresv events are run for every pbs_ralter command.
Modifyresv hooks may alter the same set of variables in the reservation event that resvsub hooks can.
Hooks registered to modifyresv will be run at the beginning of req_modifyReservation, prior to the reservation alteration being processed and applied.
pbs.event().accept() allows for the reservation modification to proceed and the changes be applied if the modification request is otherwise valid after the hook has run.
pbs.event().reject() will prevent the reservation modification from completing.

Reverser dictionary

pbs.REVERSE_RESV_STATE has been added
This set of states works for reservation_state and reservation_substate fields as both entries use values from the same set of RESV_STATEs.

Internals

RESV_BEGIN

New functional tests for resv_begin hook events.
New PBS batch request event "rq_resresvbegin" to enable hook processing.
New event type: HOOK_EVENT_RESV_BEGIN
A new batch request to allow for hook handling, the PBS_BATCH_BeginResv dummy request, has been added.
A call to process_hooks() has been added to Time4resv.
This hook reuses the reservation information structure that resv_end uses. No new data is being added at this time.
This hook will run from the server when the server processes the reservation beginning. This will not be affected by the scheduler starting the reservation's queue in it's own internal loop.
This hook runs after the scheduler has been informed of the reservation that is beginning and occurs immediately after an email may be sent to the owner and the B-accounting record is written.

RESV_CONFIRM

New functional tests for resv_confirm hook events.
New event type: HOOK_EVENT_RESV_CONFIRM
process_hooks has been extended to run resv_confirm hooks when run with the PBS_BATCH_ConfirmResv event.
This hook reuses the reservation information structure that resv_end uses. No new data is being added at this time.
A handler for HOOK_EVENT_RESV_CONFIRM has been added to _pbs_python_event_set to retrieve data from the correct reservation id as the event passed to process_hooks is a rq_runjob structure and not an rq_manage structure with the correct data, unlike the handler for resv_begin and resv_end events.
A call to process_hooks() has been added to req_confirmresv.
This hook runs whenever a reservation has been confirmed and a Y-accounting record would have been written. This should run on every confirmation and re-confirmation of a reservation.

MODIFYRESV

New functional tests to check the operation of the modifyresv hook.
New hook event type: HOOK_EVENT_MODIFYRESV
process_hooks has been extended to run modifyresv hooks when run with the PBS_BATCH_ModifyResv event.
In _pbs_python_event_set handling for the modifyresv hook event has been added. -- This also adds an internal resv_o variable similar to how job_o is used in the HOOK_EVENT_MODIFYJOB handler.
A handler for HOOK_EVENT_MODIFYRESV has been added to _pbs_python_event_to_request to translate data written to the hook's event back into the initial modify request to allow the hook to modify appropraite parameters.
A call to process_hooks() has been added to req_modifyReservation.
The reservation event has writable attributes. These are the same attributes that are supported for the resvsub hook. Since resvsub can modify these parameters in a site-specified way, modifyresv should also be able to do so to preserve consistency in behavior between pbs_rsub and pbs_ralter's spite specified behavior.
The changes in the modifyresv hook may be rejected by the server in subsequent processing even if the hook itself passes. A successful modification should also result in a reservation re-confirmation which should cause the resv_confirm hooks to be run.
The changes to the reservation may be found in pbs.event().resv. The current state prior to changes being applied may be found in pbs.event().resv_o similar to how the objects for modifyjob hooks are populated.

Notes

Existing reservation objects have been used (resvsub's data for modifyresv and resv_end for resv_begin and resv_confirm).
Existing event parameters were used with the same read/write behavior as above.
Due to reservation state being potentially updated internally to the scheduler separate from the server, resv_begin cannot safely modify a reservation once begun or it's queues or else we will definitely have a race condition when the advance reservation goes active. (see src/server/svr_jobfunc.c:Time4resv and src/scheduler/resource_resv.c:update_resresv_on_run)
rq_resresvbegin is used to to allow process_hooks to run, as a pseudo-event since there is no PBS_BATCH event that actually triggers Time4resv where this hook is started from.

OSS Site Map

Project Documentation Main Page

Developer Guide Pages

Browser not supported