Page Comparison

Objective

...

• Visibility: Public
• Change Control: Stable
• Value: 'all', 'job_start', or 'none'
• Python type: str
• Synopsis:  
  • When set to 'all', this means to tolerate all node failures resulting from communication problems (e.g. polling) between the primary mom and the sister moms assigned to the job, as well as due to rejections from execjob_begin, or execjob_prologue hook execution by remote moms.
  • When set to 'job_start', this  means to tolerate node failures that occur only during job start like an assigned sister mom failing to join job, communication errors that happen between the primary mom or sister moms, just before the job executes the execjob_launch hook and/or the top level shell or executable.

  • When set to 'none' or if the attribute is unset, this means no node failures are tolerated (default behavior).

  • It can be set via qsub, qalter, or in a Python hook say queuejob hook. If set via qalter and the job is already running, it will be consulted the next time the job is rerun.
  • This can also be specified in the server attribute 'default_qsub_arguments' to allow all jobs to be submitted with tolerate_node_failures attribute set.
  • This option is best used when job is assigned extra nodes using pbs.event().job.select.increment_chunks() method (interface 7).
  • The ‘tolerate_node_failures’ job option is currently not supported on Cray systems. If specified, a Cray primary mom would ignore the setting.
    
• Privilege: user, admin, or operator can set it
• Examples:
  • Via qsub:

...

                            qalter -W tolerate_node_failures="job_start" <jobid>

• • Via a hook:

                            # cat qjob.py
                            import pbs
                            e=pbs.event()
                            e.job.tolerate_node_failures = "all"
                            # qmgr -c "create hook qjob event=queuejob"
                            # qmgr -c "import hook application/x-python default qjob.py"
                            % qsub job.scr
                            23.borg
                            % qstat -f 23
                              ...
                              tolerate_node_failures = all

• Log/Error messages:
  • When a job that has toleranttolerate_node_failures attribute set to 'true'all' or 'job_start', the following mom_logs messages will appear in the following conditions: sister moms that failed to join job due to either communication error or execjob_begin hook rejects, when a sister mom fails to setup a job like cpuset creation failure, when a sister mom rejects an execjob_prologue hook, when the primary mom fails to poll a sister mom for status, or for any communication error to the sister mom:
    • DEBUG level: "ignoring from <node_host> error as job is tolerant of node failures"

...

• Visibility: Public
• Change Control: Stable
• Python Type: dict (dictionary of pbs.vnode objects keyed by vnode name)
• Details:
  This is a new event parameter for the execjob_prologue and execjob_launch hook. It will contain the list of vnodes and their assigned resources that are managed by unhealthy moms. This can include those vnodes from sister moms that failed to join the job, that rejected an execjob_begin hook or execjob_prologue hook request, encountered communication error while primary mom is polling the sister mom host. This dictionary object is keyed by vnode name. And one can walk through this list and start offlining the vnodes, for example:

  for vn in e.vnode_list_fail:
      v = e.vnode_list_fail[vn]
      pbs.logmsg(pbs.LOG_DEBUG, "offlining %s" % (vn,))
      v.state = pbs.ND_OFFLINE

• Additional Details:
  • Any sister nodes that are able to join the job will be considered as healthy.
  • The sucess of join job request maybe the result of a check made by a remote execjob_begin hook. After successfully joining the job, the node may further check its status via a remote execjob_prologue hook. A reject by the remote prologue hook will cause primary mom to treat the sister node as a problem node, and will be marked as unhealthy. Unhealthy nodes are not selected when pruning a job'srequest s request via the pbs. release_nodes(keep_select) call (see interface 8 below).
  • If there's an execjob_prologue hook in place, the primary mom would track node hosts that have given IM_ALL_OKAY acknowledgement for their execution of the execjob_prologue hook. Then after some ‘job_launch_delay’ amount of time of job startup (interface 4), primary mom would start reporting as failed nodes those who have not given their positive acknowledgement during prologue hook execution. This info is communicated to the child mom running on behalf of the job, so that vnodes from the failed hosts would not be used when pruning a job (i.e. pbs. release_nodes(keep_select=X) call).

  • If after some time, a node's host comes back with an acknowledgement of successful prologue hook execution, the primary mom would add back the host to the healthy list.

...

                   ...

                  pbs.event().job.release_nodes(keep_select=...)

...

• • This call makes sense only when job is node failure tolerant (i.e. toleranttolerate_node_failures=job_start or tolerate_node_failures=all) since it is when the
    list of healthy and failed nodes are gathered to be consulted by release_nodes() for determining which chunk should be assigned, freed. If it is invoked and yet the job is not tolerant of node failures, the following message is displayed in mom_logs under DEBUG level:

...

                   Seeing this log message means that a job can momentarily receive an error when doing tm_spawn or pbsdsh to a node that did not complete the nodes table update yet.

• • When mother superior fails  to prune currently assigned chunk resource, then the following detailed mom_logs message are shown in DEBUG2 level:
    

    • "could not satisfy select chunk (<resc1>=<val1> <resc2>=<val2> ...<rescN>=valN) 

    • "NEED chunks for keep_select (<resc1>=<val1> <resc2>=<val2> ...<rescN>=valN)
    • "HAVE chunks from job's exec_vnode (<exec_vnode value>

...

• Visibility: Public
• Change Control: Stable
• Python constant: pbs.EXECJOB_RESIZE
• Event Parameters: 
  • pbs.event().job - This is a pbs.job object representing the job whose resources has been updated. This job object cannot be modified under this hook.
  • pbs.event().vnode_list[] - This is a dictionary of pbs.vnode objects, keyed by vnode name, listing the vnodes that are assigned to the job. The vnode objects in the vnode_list cannot be modified.
• Restriction: The execjob_resize hook will run under the security context of Admin user.
• Details:
  • An execjob_resize event has been introduced primarily as a new event for pbs_cgroups hook to be executed when there's an update to the job's assigned resources, as a result of the release_nodes() call. This would allow pbs_cgroups to act on a change to job's resources. The action would be to update the limits of the job's cgroup.
    

  • If the pbs_cgroups hook is executing in response to an execjob_resize event,  calling pbs.event().reject(<message>),  encountering an exception, or terminating due to an alarm call, would result in the following DEBUG2 mom_logs message, and the job is aborted on the mom side, and requeued/rerun on the server side:

    “execjob_resize” request rejected by ‘pbs_cgroups”
    <message>

• New qmgr output:

  • The returned error message from qmgr upon seeing an unrecognized hook event has changed (due to this additional hook event):

    # qmgr –c “set hook <hook_name> event = <bad_event>”

    from:
    invalid argument (yaya) to event. Should be one or more of: queuejob,modifyjob,resvsub,movejob,runjob,provision,periodic,resv_end,execjob_begin,execjob_prologue,execjob_epilogue,execjob_preterm,execjob_end,exechost_periodic,execjob_launch,exechost_startup,execjob_attach or "" for no event

    to:
    invalid argument (yaya) to event. Should be one or more of: queuejob,modifyjob,resvsub,movejob,runjob,provision,periodic,resv_end,execjob_begin,execjob_prologue,execjob_epilogue,execjob_preterm,execjob_end,exechost_periodic,execjob_launch,exechost_startup,execjob_attach,execjob_resize or "" for no event

• External documentation:
  • This hook event is intentionally not added to the external documentation (as of 2021.1.3), because it is intended for use primarily by the cgroups hook.

Case of Reliable Job Startup:

...