PP-864 Support suspend/resume on Cray X* series

https://pbspro.atlassian.net/browse/PP-864

Forum discussion

Overview:

Cray X* series systems have the ability to support suspending one or more jobs to run a higher priority job.  PBS needs to modify the suspend pseudo signal (used by the qsig command and preemption) to support doing suspend and resume on a Cray X* series.

Important things to note: 

• Cray systems with a Gemini interconnect do NOT support suspend/resume
• Cray systems with an Aries interconnect and newer Cray X* series systems DO support suspend/resume 
• In order to do suspend/resume set suspendResume 1 in /etc/opt/cray/alps/alps.conf (using xtopview on CLE 5.2 and prior CLEs) and then restart ALPS
  • Please refer to Cray's System Administration Guide for more details about using suspend/resume on Cray X* series
• On Cray X* series system PBS issues a request to ALPS to switch IN (resume) or OUT (suspend) an ALPS reservation
• On a Cray X* series, the suspended low priority job and the high priority job must fit into the Cray compute node’s memory
• On a Cray X* series systems have a limitation of having at maximum of 4 co-resident jobs on a compute node. Please read Cray documentation for more details.

Interface #1 - New error code when ALPS fails to switch reservation from suspend to resume or resume to suspend

• Change Control: Stable
• Details:
  
  • New error code "15219" will be returned for pbs_sigjob() IFL call when ALPS fails to switch reservation in mom.
  • Command "qsig" will print the following error message when ALPS fails to switch reservation
    "qsig: Switching ALPS reservation failed <job id>"

Interface #2 - New mom log messages

• Change Control: Unstable
• Details:
  
  • Following mom log message is logged on Cray X* series systems when 
    • PBS tries to suspend/resume a job (PBSEVENT_DEBUG2)-
      "Switching ALPS reservation <ALPS reservation id> to <suspend/resume>"
    • ALPS fails to accept a reservation switch request (PBSEVENT_SYSTEM)-
      "Failed to switch <OUT/IN> ALPS reservation"
    • PBS issues the ALPS reservation switch request successfully (PBSEVENT_DEBUG2)-
      "Made the ALPS SWITCH request"
    • It is possible to incorrectly get an 'EMPTY' response (which means there is no claim on the ALPS reservation) when in reality there is a claim on the ALPS reservation.  PBS will print this log message so it is possible to see how often the false 'EMPTY' response is received (PBSEVENT_DEBUG2).
      "ALPS reservation <ALPS reservation ID> SWITCH status is = 'EMPTY'"

Interface #3 -Server attribute "restrict_res_to_release_on_suspend" is set to "ncpus" by default on Cray X* series systems

• Change Control: Stable
• Details:
  
  • During suspension of a job PBS will only release ncpus on Cray by default 

  • This attribute is set in pbs_habitat script which is executed when PBS is started for the first time after an install/upgrade

  • Admin may choose to update "restrict_res_to_release_on_suspend" and add more resources to it. But, removing "ncpus" from the list of resource names is not advisable on Cray X* series system (Please read this page for more information in "restrict_res_to_release_on_suspend" server attribute)

Interface #4 - Jobs with exclusive placement can not be suspended

• Change Control: Stable
• Details:
  • On a Cray X* series system a job that requests exclusive access (i.e. -lplace=excl) to a node can not be suspended. An error is thrown by ALPS while trying to switch out an exclusive job's ALPS reservation
  • If it is tried to be suspended mom returns with an error code 15219 and logs an DEBUG level error message as mentioned in Interface 5

Interface #5 - New mom log messages

• Change Control: Unstable
• Details:
  
  • Following mom log message is logged on Cray X* series systems when 
    • While switching out an exclusive ALPS reservation (suspending a job with exclusive placement) following error is logged (PBSEVENT_DEBUG)

      "BASIL;ERROR: ALPS error: apsched: at least resid <ALPS reservation id> is exclusive"