PP-388: Allow mom hooks to accumulate resources_used values

Objective

This is to enhance PBS reporting of resources_used values, in particular, have MoM accumulate resources_used values that are set in a hook, whether builtin resource or custom resource.

Interface 1: For multi-node jobs, report accumulated resources_used values in accounting logs/qstat -f output, for those resources set in a hook.

Visibility: Public
Change Control: Stable
Synopsis: Display accumulated resources_used values in accounting logs and qstat -f output, for resources that are set in an execjob_prologue, execjob_epilogue, or exechost_periodic hook.
Details:
Resources_used resources 'cput', 'mem', 'cpupercent' will continue to be aggregated and reported as before.
The additional resources that can be accumulated are those that are set in a hook, which can be a builtin resource (e.g vmem), or a custom resource.
Builtin resource: If a builtin resource is set in a hook, then any polling done (if any) by MoM for its value will automatically be discontinued. The hook then becomes in charge of updating the value.
Custom resource: For a custom resource to be set in a hook, the resource must have already been added to PBS in one of 2 ways:

Via qmgr:
^{# qmgr -c "create resource <res_name> type=<res_type>,flag=h}
Via a mom exechost_startup hook as follows:
^{# qmgr -c "create hook start event=exechost_startup"}
^{# qmgr -c "import hook start application/x-python default start.py"}
^{# qmgr -c "export hook start application/x-python default"}
^{import pbs}
^{e=pbs.event()}
^{localnode=pbs.get_local_nodename()}
^{e.vnode_list[localnode].resources_available['foo_i'] = 7}
^{e.vnode_list[localnode].resources_available['foo_f'] = 5.0}
^{e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven"}
,
Aggregation of values: The resource value collected in mother superior mom is aggregated with each of the values obtained from the sister moms whose nodes are part of the job.
For resources of type float, long, and size, the value will be reported in accounting logs and qstat -f as:
resources_used.<resource_name> = <summed total>
If for some reason a sister node did not report back the resources_used value for the resource, then the last know value will be used.
For resources of type string, the value is aggregated on a per-MOM basis.
The value obtained from each MOM must be a valid JSON object (a Python dictionary), which is an unordered set of name/value pairs, where each object begins with { (left brace) and ends with } (right brace). Each name is followed by: (colon) and the name/value pairs are separated by , (comma). The name must be wrapped in double quotes allowing backslash escapes.
When all values are found to be of valid JSON format, then the resulting string resource value would be a merging (i.e. union) of all dictionary items, and shown in qstat -f and accounting_logs as:
resources_used.<resource_name> = { <momA_JSON_item_value>, <momB_JSON_item_value>, <momC_JSON_item_value>, ..}
Ex. if momA returned '{ "a":1, "b":2 }', momB returned '{ "c":1 }', and momC returned '{"d":4}' for resources_used.foo_str, then we get:

resources_used.foo_str='{"a": 1, "b": 2, "c":1,"d": 4}'

NOTE:If 2 or more values have the same 'name' as key, then one of them will be retained, which will depend on Python's operation of merging dictionary items. It is recommended for hook writers to make the keys unique, and this can be done by using the pbs.get_local_nodename() value as part of the key.
When at least one of the values obtained from a sister MOM is not of JSON format, then the string cannot be accumulated, resulting in an unset resources_used string value. There'll be an error message in mom_logs that will be reported as follows: " Job <jobid> resources_used.<string_resource> cannot be accumulated: value <input value> from mom <hostname> not JSON-format: <exception_error_message>."
Examples:
Given an epilogue hook that runs on all the mom nodes, setting different resources_used values based on whether executing on a MS mom or sister mom:
.#: qmgr -c "list hook epi"
Hook epi
type = site
enabled = true
event = execjob_epilogue
user = pbsadmin
alarm = 30
order = 1
debug = false
fail_action = none
# qmgr -c "e h epi application/x-python default"
import pbs
e=pbs.event()
pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook")
if e.job.in_ms_mom(): #set in MS mom
e.job.resources_used["vmem"] = pbs.size("9gb")
e.job.resources_used["foo_i"] = 9
e.job.resources_used["foo_f"] = 0.09
e.job.resources_used["foo_str"] = '{"nine":9}'
e.job.resources_used["cput"] = 10
e.job.resources_used["foo_assn2"] = '{"vn1":1,"vn2":2,"vn3":3}'
else: # set in sister mom
e.job.resources_used["vmem"] = pbs.size("10gb")
e.job.resources_used["foo_i"] = 10
e.job.resources_used["foo_f"] = 0.10
e.job.resources_used["foo_str"] = '{"ten":10}'
e.job.resources_used["cput"] = 20
e.job.resources_used["foo_assn2"] = '{"vn4":4,"vn5":5,"vn6":6}'
Now with 2 nodes:
Submit the following job:
% cat job.scr2
PBS -l select=2:ncpus=1
pbsdsh -n 1 hostname
sleep 300

% qsub job.scr2
102.corretja
When the job completes, the following resources_used values are shown:
With server job_history_enabled=true, one can check the values in a finished job:
% qstat -x -f 102
...
resources_used.cpupercent = 0
resources_used.cput = 00:00:30
resources_used.vmem = 19gb
resources_used.foo_f = 0.19
resources_used.foo_i = 19
resources_used.foo_str = '{"nine": 9, "ten": 10}'
resources_used.foo_assn2='{"vn1": 1, "vn2": 2 ,"vn3": 3 ,"vn4": 4, "vn5": 5, "vn6": 6}'
resources_used.mem = 0kb
resources_used.ncpus = 2
resources_used.walltime = 00:00:05

NOTE: Those in bold show values accumulated between the MS value and the sister value.
The accounting_logs show the same values:
8/03/2016 18:28:13;E;102.corretja;user=alfie group=users project=_pbs_project_default jobname=job.scr2 queue=workq ctime=1470263288 qtime=1470263288 etime=1470263288 start=1470263288 exec_host=corretja/0+nadal/0 exec_vnode=(corretja:ncpus=1)+(nadal:ncpus=1) Resource_List.ncpus=2 Resource_List.nodect=2 Resource_List.place=free Resource_List.select=2:ncpus=1 session=16986 end=1470263293 Exit_status=143 resources_used.cpupercent=0 resources_used.cput=00:00:30 resources_used.vmem=19gb resources_used.foo_f=0.19 resources_used.foo_i=19 resources_used.foo_str='{"nine": 9, "ten": 10}' resources_used.foo_assn2='{"vn1": 1, "vn2": 2 ,"vn3": 3 ,"vn4": 4, "vn5": 5, "vn6": 6}' resources_used.mem=0kb resources_used.ncpus=2 resources_used.walltime=00:00:05 run_count=1
Now supposed that I change the execjob_epilogue hook to only set resources_used values from the MS mom:
# qmgr -c "e h epi application/x-python default"
import pbs
e=pbs.event()
pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook")
if e.job.in_ms_mom():
e.job.resources_used["vmem"] = pbs.size("9gb")
e.job.resources_used["foo_i"] = 9
e.job.resources_used["foo_f"] = 0.09
e.job.resources_used["foo_str"] = '{"nine":9}'
e.job.resources_used["cput"] = 10
Then submit the job and then deleting it to force execjob_epilogue hook execution, resulted in:
% qsub job.scr2
103.corretja

% qstat
Job id Name User Time Use S Queue
---------------- ---------------- ---------------- -------- - -----
103.corretja job.scr2 alfie 00:00:00 R workq

% qdel 103

% qstat -f -x 103
Job Id: 103.corretja
Job_Name = job.scr2
Job_Owner = alfie@corretja
resources_used.cpupercent = 0
resources_used.cput = 00:00:10
resources_used.vmem = 9gb
resources_used.foo_f = 0.09
resources_used.foo_i = 9
resources_used.foo_str = '{"nine":9}'
resources_used.mem = 0kb
resources_used.ncpus = 2
resources_used.walltime = 00:00:06
NOTE: Since this is a multinode job, but the sister mom did not report a value to 'foo_str', then only the mother superior mom value is aggregated as is.
Accounting logs show:
08/03/2016 18:36:14;E;103.corretja;user=alfie group=users project=_pbs_project_default jobname=job.scr2 queue=workq ctime=1470263768 qtime=1470263768 etime=1470263768 start=1470263768 exec_host=corretja/0+nadal/0 exec_vnode=(corretja:ncpus=1)+(nadal:ncpus=1) Resource_List.ncpus=2 Resource_List.nodect=2 Resource_List.place=free Resource_List.select=2:ncpus=1 session=17114 end=1470263774 Exit_status=143 resources_used.cpupercent=0 resources_used.cput=00:00:10 resources_used.vmem=9gb resources_used.foo_f=0.09 resources_used.foo_i=9 resources_used.foo_str='{"nine": 9}' resources_used.mem=0kb resources_used.ncpus=2 resources_used.walltime=00:00:06 run_count=1

NOTE: In hooks, when specifying values to string_array type of resources (e.g. Resource_List, resources_used), if value is enclosed in single quotes ('), then everything inside the single quotes are taken in as the value. For example,

pbs.event().job.Resource_List["foo_stra"] = '"glad,elated"' ← note of the outer single quote 'foo_stra' resource will have a value of "glad,elated"
This change was needed in order to be consistent with Python's handling of string literal values so as to align with json string processing.

Interface 2: For single-node jobs, report json string resources_used values in accounting logs/qstat -f output within single quotes, for those resources set in a hook.

Visibility: Public
Change Control: Stable
Synopsis: Display accumulated resources_used values in accounting logs and qstat -f output, for resources that are set in an execjob_prologue, execjob_epilogue, or exechost_periodic hook.
Details:
- In case of a single node job, for the resources of type string following would be applicable:
  - If the value obtained from the MOM is a valid JSON object (a Python dictionary), then the resulting string resource value would be shown in qstat -f and accounting_logs within single quote as:
    resources_used.<resource_name> = '{ <mom_JSON_item_value>, <mom_JSON_item_value>, <mom_JSON_item_value>, ..}
    Ex. if mom returned { "a":1, "b":2, "c":1,"d": 4} for resources_used.foo_str, then we get:
    resources_used.foo_str='{"a": 1, "b": 2, "c":1,"d": 4}'
  - If the value obtained from MOM is not a valid JSON, then it is reported as is in qstat -f and accounting logs.
    Ex. if mom returned "hello" for resources_used.foo_str, then we get:
    resources_used.foo_str="hello"
Community Discussion