Jobs in queues with resources_available limits (e.g., resvs) may not run due to equiv classes

Description

The scenarios is we have a standing reservation which overlaps with another reservation. When the standing reservation starts, the jobs submitted to it don't start immediately even if resources are available.

Steps to reproduce:
1. In a single node step up, set available ncpus to 4.
2. submit the following reservations:

a. overlapping reservations:
R0: $ pbs_rsub -R now+2m -E now+6m
S1: $ pbs_rsub -R now+4m -E now+8m -r "FREQ=HOURLY;COUNT=3"

b. non-overlapping reservations:
R2: $ pbs_rsub -R now+10m -E now+12m
S3: $ pbs_rsub -R now+14m -E now+16m -r "FREQ=HOURLY;COUNT=3"

3. submit jobs

a. before R0 starts
Submit 4 jobs say J4, J5, J6 & J7 .
qsub – /bin/sleep 999
Expected outcome --> J4 & J5 in R state, J6 & J7 in Q state

Submit 2 jobs per reservation as follows.

J8 & J9 to R0 : qsub -q R0 – /bin/sleep 999
Expected outcome --> J8 & J9 in Q state

J10 & J11 to S1: qsub -q S1 – /bin/sleep 999
Expected outcome --> J10 & J11 in Q state

J12 & J13 to R2 : qsub -q R2 – /bin/sleep 999
Expected outcome --> J12 & J13 in Q state

J14 & J15 to S3 : qsub -q S3 – /bin/sleep 999
Expected outcome --> J14 & J15 in Q state

b. after R0 in R state

J4, J5, J8 in R state and the rest of the jobs are in Q state

c. after S1 in R state (start of overlapping period)

J4, J5, J8, J10 in R state and the rest of the jobs are in Q state

We do not get the expected outcome in case of the overlapping standing reservation S1. It is expected that job J10 should run when reservation S1 starts. But the behavior observed in step 3c (refer steps to reproduce mentioned above ) is J4, J5, J8 are in R state when reservation S1 starts while J10 remains in Q state.

Acceptance Criteria

None

Activity

Show:
Scott Campbell
October 31, 2017, 3:22 PM

I reproduced this to see what the scheduler's comment about the job was, it is " Not Running: Insufficient amount of queue resource: ncpus (R: 1 A: 0 T: 1)":

1 node with 4 ncpus configured:

source /etc/pbs.conf
export PBS_TZID=America/Chicago
R0=$(pbs_rsub -R $(date --date=@$(echo $(($(date +%s) +120))) +%H%M) -D240 | awk -F. '{print $1}')
S1=$(pbs_rsub -R $(date --date=@$(echo $(($(date +%s) +240))) +%H%M) -D240 -r "FREQ=HOURLY;COUNT=3" | awk -F. '{print $1}')
R2=$(pbs_rsub -R $(date --date=@$(echo $(($(date +%s) +600))) +%H%M) -D120 | awk -F. '{print $1}')
S3=$(pbs_rsub -R $(date --date=@$(echo $(($(date +%s) +840))) +%H%M) -D120 -r "FREQ=HOURLY;COUNT=3" | awk -F. '{print $1}')
sleep 3
J4=$(echo sleep 99999 | qsub)
J5=$(echo sleep 99999 | qsub)
J6=$(echo sleep 99999 | qsub)
J7=$(echo sleep 99999 | qsub)
J8=$(echo sleep 99999 | qsub -q $R0)
J9=$(echo sleep 99999 | qsub -q $R0)
J10=$(echo sleep 99999 | qsub -q $S1)
J11=$(echo sleep 99999 | qsub -q $S1)
J12=$(echo sleep 99999 | qsub -q $R2)
J13=$(echo sleep 99999 | qsub -q $R2)
J14=$(echo sleep 99999 | qsub -q $S3)
J15=$(echo sleep 99999 | qsub -q $S3)

pbs_version = 18.2.0.20171027011411

[user1@centos7-2 ~]$ echo $J10
101.centos7-2

[user1@centos7-2 ~]$ echo $S1
S92

reason for J10 not running:

Not Running: Insufficient amount of queue resource: ncpus (R: 1 A: 0 T: 1)

[user1@centos7-2 ~]$ pbs_rstat S92
Resv ID Queue User State Start / Duration / End
---------------------------------------------------------------------
S92.centos S92 user1@ce RN Today 11:12 / 240 / Today 11:16
[user1@centos7-2 ~]$ date ; qstat -ns 101
Tue Oct 31 11:12:21 EDT 2017

centos7-2:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ — — ------ ----- - -----
101.centos7-2 user1 S92 STDIN – 1 1 – – Q –

Not Running: Insufficient amount of queue resource: ncpus (R: 1 A: 0 T: 1)

[root@centos7-2 PBSPro_18.2.0]# qmgr -c "p q S92"
#

  1. Create queues and set their attributes.
    #
    #

  2. Create and define queue S92
    #
    create queue S92
    set queue S92 queue_type = Execution
    set queue S92 acl_user_enable = True
    set queue S92 acl_users = user1@centos7-2.prog.altair.com
    set queue S92 resources_max.ncpus = 1
    set queue S92 resources_available.ncpus = 1
    set queue S92 enabled = True
    set queue S92 started = True
    qmgr -c "p q S92"

[root@centos7-2 PBSPro_18.2.0]# pbs_rstat -f S92
Resv ID: S92.centos7-2
Reserve_Name = NULL
Reserve_Owner = user1@centos7-2.prog.altair.com
reserve_type = 2
reserve_state = RESV_RUNNING
reserve_substate = 5
reserve_start = Tue Oct 31 11:12:00 2017
reserve_end = Tue Oct 31 11:16:00 2017
reserve_duration = 240
queue = S92
Resource_List.ncpus = 1
Resource_List.nodect = 1
Resource_List.select = 1:ncpus=1
Resource_List.place = free
Resource_List.walltime = 00:04:00
schedselect = 1:ncpus=1
resv_nodes = (centos7-2:ncpus=1)
Authorized_Users = user1@centos7-2.prog.altair.com
server = centos7-2
ctime = Tue Oct 31 11:08:41 2017
mtime = Tue Oct 31 11:08:41 2017
Variable_List = PBS_O_LOGNAME=user1,PBS_O_HOST=centos7-2.prog.altair.com,PBS_O_MAIL=/var/spool/mail/user1,PBS_TZID=America/Chicago
euser = user1
egroup = user1
reserve_rrule = FREQ=HOURLY;COUNT=3
reserve_index = 1
reserve_count = 3

[user1@centos7-2 ~]$ pbs_rstat S92
Resv ID Queue User State Start / Duration / End
---------------------------------------------------------------------
S92.centos S92 user1@ce RN Today 11:12 / 240 / Today 11:16

[user1@centos7-2 ~]$ date ; qstat -ns 101
Tue Oct 31 11:12:21 EDT 2017

centos7-2:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ — — ------ ----- - -----
101.centos7-2 user1 S92 STDIN – 1 1 – – Q –

Not Running: Insufficient amount of queue resource: ncpus (R: 1 A: 0 T: 1)

filter 0 sched logs of the cycle when the reservation started attached.

Scott Campbell
October 31, 2017, 3:35 PM
Edited

Since JIRA is mangling my formatting, steps to reproduce attached, it will create the reservations with the right time based on now. Assumes 1 node with 4 ncpus.

Bhroam Mann
November 16, 2017, 9:28 PM

Git bisect is the coolest thing since sliced bread. It found the commit which introduced this issue. The job equivalence class commit introduced this.

Bhroam Mann
November 17, 2017, 12:47 AM

This affects more than just reservations. It has to do with queues that have resources_available limits set. All reservations have resources_available limits set. Jobs in different queues can validly be placed into the same equivalence class. If there is nothing special about two queues, queue doesn't matter. In the case of resources_available limits, they do.

A much smaller steps to reproduce is the following:
Assumes node has >= 2 cpus
1. Create an additional queue workq2
2. qmgr -c 'set queue workq resources_available.ncpus=1'
3. Submit J1-J2: qsub -l ncpus=1 – /bin/sleep 1000
4. Submit J3: qsub -l ncpus=1 -q workq2 – /bin/sleep 1000

J2 can't run due to the resources_available.ncpus limit. J3 should be able to run since there is no resources_available.ncpus limit. It does not start because its equivalence class is marked as can not run.

Queue should be part of a jobs equivalence class if either it is part of a reservation, or its queue has resources_available set.

Assignee

Bhroam Mann

Reporter

Neha Padole

Severity

3-High

OS

None

Start Date

None

Pull Request URL

None

Story Points

1

Affects versions

Priority

Medium
Configure