Page Comparison

Objective

...

• Visibility: Public
• Change Control: Stable
• Value: 'all', 'job_start', or 'none'
• Python type: str
• Synopsis:  
  • When set to 'all', this means to tolerate all node failures resulting from communication problems (e.g. polling) between the primary mom and the sister moms assigned to the job, as well as due to rejections from execjob_begin, or execjob_prologue hook execution by remote moms.
  • When set to 'job_start', this  means to tolerate node failures that occur only during job start like an assigned sister mom failing to join job, communication errors that happen between the primary mom or sister moms, just before the job executes the execjob_launch hook and/or the top level shell or executable.

  • When set to 'none' or if the attribute is unset, this means no node failures are tolerated (default behavior).

  • It can be set via qsub, qalter, or in a Python hook say queuejob hook. If set via qalter and the job is already running, it will be consulted the next time the job is rerun.
  • This can also be specified in the server attribute 'default_qsub_arguments' to allow all jobs to be submitted with tolerate_node_failures attribute set.
  • This option is best used when job is assigned extra nodes using pbs.event().job.select.increment_chunks() method (interface 7).
  • The ‘tolerate_node_failures’ job option is currently not supported on Cray systems. If specified, a Cray primary mom would ignore the setting.
    
• Privilege: user, admin, or operator can set it
• Examples:
  • Via qsub:

...

                            qalter -W tolerate_node_failures="job_start" <jobid>

• • Via a hook:

                            # cat qjob.py
                            import pbs
                            e=pbs.event()
                            e.job.tolerate_node_failures = "all"
                            # qmgr -c "create hook qjob event=queuejob"
                            # qmgr -c "import hook application/x-python default qjob.py"
                            % qsub job.scr
                            23.borg
                            % qstat -f 23
                              ...
                              tolerate_node_failures = all

...

• Examples:

         Given an execjob_prologue hook, a hook writer can release a set of nodes from a job by doing:

              pj = e.job.release_nodes(keep_select="ncpus=2:mem=2gb+ncpus=2:mem=2gb+ncpus=1:mem=1gb")
              if pj != None:
                  pbs.logmsg(pbs.LOG_DEBUG, "pj.exec_vnode=%s" % (pj.exec_vnode,))
              else:               # returned None job object, so we can put a hold on the job and requeue it, rejecting the hook event
                  e.job.Hold_Types = pbs.hold_types("s")
                  e.job.rerun()
                  e.reject("unsuccessful at LAUNCH")

  
  

• Log/Error messages:
  • When job's assigned nodes get pruned (nodes released to satisfy 'keep_select') , mom_logs will show the following info under PBSEVENT_JOB log level:
    

    ";Job;<jobid>;pruned from exec_vnode=<original value>"
    ";Job;<jobid>;pruned to exec_nodevnode=<new value>"
    
    

  • When a multinode job's assigned resources have been modified, primary mom will do a quick 5 seconds wait  for an acknowledgement from the sister moms that they have updated their nodes table. When not all acknowledgements were received by primary mom during that 5 seconds wait, then there'll be this DEBUG2 level mom_logs message:

...

                   Seeing this log message means that a job can momentarily receive an error when doing tm_spawn or pbsdsh to a node that did not complete the nodes table update yet.

• • When mother superior fails  to prune currently assigned chunk resource, then the following detailed mom_logs message are shown in DEBUG2 level:
    

    • "could not satisfy select chunk (<resc1>=<val1> <resc2>=<val2> ...<rescN>=valN) 

    • "NEED chunks for keep_select (<resc1>=<val1> <resc2>=<val2> ...<rescN>=valN)
    • "HAVE chunks from job's exec_vnode (<exec_vnode value>

...