JOBOBIT Hook Event

Links

• Link to discussion on Developer Forum: https://community.openpbs.org/t/design-document-for-endjob-hook-event/2525
• Link to ALCF development repo: https://github.com/ericpershey/pbspro/compare/hook_event_jobobit
• Link to pull requests: 
  • https://github.com/openpbs/openpbs/pull/2290
  • https://github.com/openpbs/openpbs/pull/2494

Overview

In order to provide more detailed accounting of jobs as required by some sites, recording the job information and the time of when a job leaves execution is required. Therefore an jobobit hook event has been added to the PBS server. Registered jobobit hook scripts are executed when a job or subjob leaves execution. Each script is provided information about the job and any associated attributes.

Technical Details

Added Python constants:

• pbs.HOOK_EVENT_QUEUEJOB
• pbs.HOOK_EVENT_MODIFYJOB
• pbs.HOOK_EVENT_RESVSUB
• pbs.HOOK_EVENT_MODIFYRESV
• pbs.HOOK_EVENT_MOVEJOB
• pbs.HOOK_EVENT_RUNJOB
• pbs.HOOK_EVENT_JOBOBIT
• pbs.HOOK_EVENT_MANAGEMENT
• pbs.HOOK_EVENT_MODIFYVNODE
• pbs.HOOK_EVENT_PROVISION
• pbs.HOOK_EVENT_RESV_END
• pbs.HOOK_EVENT_RESV_BEGIN
• pbs.HOOK_EVENT_RESV_CONFIRM
• pbs.HOOK_EVENT_EXECJOB_BEGIN
• pbs.HOOK_EVENT_EXECJOB_PROLOGUE
• pbs.HOOK_EVENT_EXECJOB_EPILOGUE
• pbs.HOOK_EVENT_EXECJOB_PRETERM
• pbs.HOOK_EVENT_EXECJOB_END
• pbs.HOOK_EVENT_EXECJOB_LAUNCH
• pbs.HOOK_EVENT_EXECHOST_PERIODIC
• pbs.HOOK_EVENT_EXECHOST_STARTUP
• pbs.HOOK_EVENT_EXECJOB_ATTACH
• pbs.HOOK_EVENT_EXECJOB_RESIZE
• pbs.HOOK_EVENT_EXECJOB_ABORT
• pbs.HOOK_EVENT_EXECJOB_POSTSUSPEND
• pbs.HOOK_EVENT_EXECJOB_PRERESUME
• pbs.HOOK_EVENT_MOM_EVENTS
• pbs.HOOK_EVENT_PERIODIC

• pbs.REVERSE_JOB_STATE
• pbs.REVERSE_JOB_SUBSTATE
• pbs.REVERSE_HOOK_EVENT

• pbs.JOB_SUBSTATE_UNKNOWN
• pbs.JOB_SUBSTATE_TRANSIN
• pbs.JOB_SUBSTATE_TRANSICM
• pbs.JOB_SUBSTATE_TRNOUT
• pbs.JOB_SUBSTATE_TRNOUTCM
• pbs.JOB_SUBSTATE_QUEUED
• pbs.JOB_SUBSTATE_PRESTAGEIN
• pbs.JOB_SUBSTATE_SYNCRES
• pbs.JOB_SUBSTATE_STAGEIN
• pbs.JOB_SUBSTATE_STAGEGO
• pbs.JOB_SUBSTATE_STAGECMP
• pbs.JOB_SUBSTATE_HELD
• pbs.JOB_SUBSTATE_SYNCHOLD
• pbs.JOB_SUBSTATE_DEPNHOLD
• pbs.JOB_SUBSTATE_WAITING
• pbs.JOB_SUBSTATE_STAGEFAIL
• pbs.JOB_SUBSTATE_PRERUN
• pbs.JOB_SUBSTATE_RUNNING
• pbs.JOB_SUBSTATE_SUSPEND
• pbs.JOB_SUBSTATE_SCHSUSP
• pbs.JOB_SUBSTATE_EXITING
• pbs.JOB_SUBSTATE_STAGEOUT
• pbs.JOB_SUBSTATE_STAGEDEL
• pbs.JOB_SUBSTATE_EXITED
• pbs.JOB_SUBSTATE_ABORT
• pbs.JOB_SUBSTATE_KILLSIS
• pbs.JOB_SUBSTATE_RUNEPILOG
• pbs.JOB_SUBSTATE_OBIT
• pbs.JOB_SUBSTATE_TERM
• pbs.JOB_SUBSTATE_DELJOB
• pbs.JOB_SUBSTATE_RERUN
• pbs.JOB_SUBSTATE_RERUN1
• pbs.JOB_SUBSTATE_RERUN2
• pbs.JOB_SUBSTATE_RERUN3
• pbs.JOB_SUBSTATE_EXPIRED
• pbs.JOB_SUBSTATE_BEGUN
• pbs.JOB_SUBSTATE_PROVISION
• pbs.JOB_SUBSTATE_WAITING_JOIN_JOB
• pbs.JOB_SUBSTATE_TERMINATED
• pbs.JOB_SUBSTATE_FINISHED
• pbs.JOB_SUBSTATE_FAILED
• pbs.JOB_SUBSTATE_MOVED

Details:

• The jobobit event will contain job information
• The job object attached to the the jobobit event is read-only
• Type for jobobit event is pbs.HOOK_EVENT_JOBOBIT
• jobobit hooks run by the server
• Hooks registered to the jobobit event will execute when:
• a job ends
  • A call to process_hooks() has been added to the end_job function (req_jobobit.c)
• a job array parent jobends
  • A call to process_hooks() has been added to the chk_array_doneness function (array_func.c)
• a subjob ends
  • A call to process_hooks() has been added to the end_job function (req_jobobit.c)
• a job needs to rerun
  • A call to process_hooks() has been added to the force_reque function (req_rerun.c)
  • A call to process_hooks() has been added to the on_job_rerun function (req_jobobit.c)
• a running job is deleted and the mom can not be accessed
  • A call to process_hooks() has been added to the post_discard_job function (node_manager.c)
• a running job is forced deleted
  • A call to process_hooks() has been added to the req_deletejob2 function (req_delete.c)

• New job attribute "obittime"
• Format: seconds since epoch time
• Permissions: read-only for all users
• Represents the time a job or subjob obit occurs (state 'F')
• A reservation object is attached to the job object if a reservation is associated with the job
• A pbs.event().accept() call terminates hook execution, as does pbs.event().reject(). The jobobit event object is unaffected by either call. 
  • In the case where the hook script encounters an exception, the error is logged.
• New functional test for jobobit hook event defined in pbs_hook_jobobit.py, includes tests for:
  • single jobs
  • array jobs
  • jobs run under reservations
  • jobs that are requeued
• New code for jobobit hook event has been added, including:
  • New batch request structure:

/* JobObit */
struct rq_jobobit {
    struct job *rq_pjob;
    char rq_jid[PBS_MAXSVRJOBID + 1];
    char *rq_destin;
};

• New event type

HOOK_EVENT_JOBOBIT

Example Hook Script for jobobit:

import pbs
import sys
try:
    e = pbs.event()
    job = e.job
    pbs.logjobmsg(job.id, 'jobobit hook, "%s" started' % (e.hook_name,))
    pbs.logjobmsg(job.id, 'jobobit hook, job starttime:%s' % (job.stime,))
    pbs.logjobmsg(job.id, 'jobobit hook, job obittime:%s' % (job.obittime,))
    pbs.logjobmsg(job.id, 'jobobit hook, job_state=%s' % (job.job_state,))
    pbs.logjobmsg(job.id, 'jobobit hook, job_substate=%s' % (job.substate,))
    state_desc = pbs.REVERSE_JOB_STATE.get(job.job_state, '(None)')
    substate_desc = pbs.REVERSE_JOB_SUBSTATE.get(job.substate, '(None)')
    pbs.logjobmsg(job.id, 'jobobit hook, job_state_desc=%s' % (state_desc,))
    pbs.logjobmsg(job.id, 'jobobit hook, job_substate_desc=%s' % (substate_desc,))
    if hasattr(job, "resv") and job.resv:
        pbs.logjobmsg(job.id, 'jobobit hook, resv:%s' % (job.resv.resvid,))
        pbs.logjobmsg(job.id, 'jobobit hook, resv_nodes:%s' % (job.resv.resv_nodes,))
        pbs.logjobmsg(job.id, 'jobobit hook, resv_state:%s' % (job.resv.reserve_state,))
    else:
        pbs.logjobmsg(job.id, 'jobobit hook, resv:(None)')
    pbs.logjobmsg(job.id, 'jobobit hook, "%s" finished' % (e.hook_name,))
except Exception as err:
    ty, _, tb = sys.exc_info()
    pbs.logmsg(pbs.LOG_ERROR, "jobobit hook, error: " + str(ty) +
        str(tb.tb_frame.f_code.co_filename) + str(tb.tb_lineno))
    e.reject()
else:
    e.accept()

PBS server log excerpt of an jobobit hook executing:

10/11/2021 19:57:16.179481;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, "jobobit_example" started
10/11/2021 19:57:16.179498;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, job starttime:1633982235
10/11/2021 19:57:16.179505;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, job obittime:1633982236
10/11/2021 19:57:16.179511;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, job_state=5
10/11/2021 19:57:16.179521;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, job_substate=53
10/11/2021 19:57:16.179529;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, job_state_desc=JOB_STATE_EXITING
10/11/2021 19:57:16.179534;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, job_substate_desc=JOB_SUBSTATE_EXITED
10/11/2021 19:57:16.179540;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, resv:(None)
10/11/2021 19:57:16.179545;0008;Server@pdw-s1;Job;7.pdw-s1;jobobit hook, "jobobit_example" finished

OSS Site Map

Project Documentation Main Page

Developer Guide Pages