2018-10-10 07:17:56,198 INFO input command: pbs_benchpress -p nomom=server,moms=mom@/etc/pbs.conf,momtype=mom@cpuset -t TestSchedSubjobBadstate -o /tmp/TestSchedSubjobBadstate.txt 2018-10-10 07:17:56,208 INFO param: nomom=server,moms=mom@/etc/pbs.conf,momtype=mom@cpuset 2018-10-10 07:17:56,212 INFO ptl version: 19.2.0 2018-10-10 07:17:56,217 INFO platform: Linux server 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 2018-10-10 07:17:56,221 INFO python version: 2.7.13 2018-10-10 07:17:56,226 INFO user: root 2018-10-10 07:17:56,229 INFO -------------------------------------------------------------------------------- 2018-10-10 07:17:56,233 INFO Cleaning up temporary files 2018-10-10 07:17:56,247 INFO Cleaning up /var/tmp dir 2018-10-10 07:17:56,254 INFO Cleaning up /tmp dir 2018-10-10 07:18:28,435 INFO ====================================================================== 2018-10-10 07:18:28,440 INFO suite name: TestSchedSubjobBadstate 2018-10-10 07:18:28,444 INFO ====================================================================== 2018-10-10 07:18:28,450 INFO =========================================== 2018-10-10 07:18:28,453 INFO Entered TestSchedSubjobBadstate setUpClass 2018-10-10 07:18:28,457 INFO =========================================== 2018-10-10 07:18:28,463 INFOCLI2 server: id pbsuser 2018-10-10 07:18:28,551 INFOCLI2 server: id pbsuser1 2018-10-10 07:18:28,632 INFOCLI2 server: id pbsuser2 2018-10-10 07:18:28,714 INFOCLI2 server: id pbsuser3 2018-10-10 07:18:28,815 INFO FQDN name server.ib0.smc-default.chf.rdlabs.hpecorp.net differs from name provided server 2018-10-10 07:18:29,112 INFO server server: server operating mode set to cli 2018-10-10 07:18:29,120 INFOCLI server: /opt/pbs/bin/qstat -Bf server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:29,425 INFO server server: version 19.2.0 2018-10-10 07:18:29,434 INFO expect action: created new action kicksched 2018-10-10 07:18:29,439 INFO expect action: added action kicksched to server server 2018-10-10 07:18:29,447 INFO FQDN name server.ib0.smc-default.chf.rdlabs.hpecorp.net differs from name provided server 2018-10-10 07:18:29,573 INFO FQDN name server.ib0.smc-default.chf.rdlabs.hpecorp.net differs from name provided server 2018-10-10 07:18:29,975 INFOCLI2 server: sudo -H /opt/pbs/sbin/pbsfs 2018-10-10 07:18:30,384 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2018-10-10 07:18:30,624 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2018-10-10 07:18:30,866 INFOCLI server: /opt/pbs/bin/qmgr -c list sched default 2018-10-10 07:18:31,553 INFOCLI2 server: sudo -H /opt/pbs/sbin/pbsfs 2018-10-10 07:18:31,937 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2018-10-10 07:18:32,158 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2018-10-10 07:18:32,390 INFO FQDN name mom.ib0.smc-default.chf.rdlabs.hpecorp.net differs from name provided mom 2018-10-10 07:18:42,794 INFO ============================================ 2018-10-10 07:18:42,801 INFO Completed TestSchedSubjobBadstate setUpClass 2018-10-10 07:18:42,805 INFO ============================================ 2018-10-10 07:18:42,819 INFO test name: test_sched_badstate_subjob (tests.functional.pbs_sched_subjob_badstate.TestSchedSubjobBadstate)... 2018-10-10 07:18:42,826 INFO test start time: Wed Oct 10 07:18:42 2018 2018-10-10 07:18:42,831 INFO test docstring: This test case tests if scheduler goes into infinite loop when following conditions are met. - Kill a mom - mark the mom's state as free - submit an array job - check the sched log for "Leaving sched cycle" from the time array job was submitted. If we are unable to find a log match then scheduler is in endless loop and test case has failed. 2018-10-10 07:18:42,839 INFO ====================================== 2018-10-10 07:18:42,844 INFO Entered TestSchedSubjobBadstate setUp 2018-10-10 07:18:42,849 INFO ====================================== 2018-10-10 07:18:42,857 INFOCLI server: /opt/pbs/bin/qstat -Bf server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:43,162 INFO status on server: server 2018-10-10 07:18:43,171 INFOCLI server: /opt/pbs/bin/qstat -Bf server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:43,469 INFO manager on server: unset server managers 2018-10-10 07:18:43,479 INFOCLI server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2018-10-10 07:18:43,996 INFOCLI server: /opt/pbs/bin/qstat -Bf server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:44,296 INFO expect on server server: managers unset server server.ib0.smc-default.chf.rdlabs.hpecorp.net ... OK 2018-10-10 07:18:44,306 INFO manager on server: set server {'managers': (2, 'root@*')} 2018-10-10 07:18:44,313 INFOCLI server: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2018-10-10 07:18:44,805 INFO server server: reverting configuration to defaults 2018-10-10 07:18:44,817 INFOCLI server: /opt/pbs/bin/qstat -Bf server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:45,118 INFO select on server: __ALL__ 2018-10-10 07:18:45,127 INFOCLI server: /opt/pbs/bin/qselect 2018-10-10 07:18:45,425 INFOCLI server: /opt/pbs/bin/qstat -f @server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:45,733 INFO expect on server server: job_state set 0 job ... OK 2018-10-10 07:18:45,744 INFOCLI server: /opt/pbs/bin/pbs_rstat -f 2018-10-10 07:18:46,044 INFO manager on server: unset server ['comment'] 2018-10-10 07:18:46,055 INFOCLI server: /opt/pbs/bin/qmgr -c unset server comment 2018-10-10 07:18:46,398 INFOCLI server: sudo -H /opt/pbs/bin/qmgr -c list hook 2018-10-10 07:18:46,864 INFOCLI server: /opt/pbs/bin/qstat -Qf @server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:47,164 INFO status on server: node 2018-10-10 07:18:47,174 INFOCLI server: /opt/pbs/bin/pbsnodes -s server.ib0.smc-default.chf.rdlabs.hpecorp.net -v -a 2018-10-10 07:18:47,635 INFO manager on server: delete queue workq 2018-10-10 07:18:47,643 INFOCLI server: /opt/pbs/bin/qmgr -c delete queue workq 2018-10-10 07:18:47,947 INFO server server: expect offset set to 0.5 2018-10-10 07:18:48,459 INFOCLI server: /opt/pbs/bin/qstat -Qf workq@server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:48,746 INFO expect on server server: unset queue workq ... OK 2018-10-10 07:18:48,755 INFO manager on server: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2018-10-10 07:18:48,763 INFOCLI server: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2018-10-10 07:18:49,075 INFO status on server: queue workq 2018-10-10 07:18:49,084 INFOCLI server: /opt/pbs/bin/qstat -Qf workq@server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:49,383 INFO server server: expect offset set to 0.5 2018-10-10 07:18:49,894 INFOCLI server: /opt/pbs/bin/qstat -Qf workq@server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:50,196 INFO expect on server server: started set True || queue_type set Execution || enabled set True queue workq ... OK 2018-10-10 07:18:50,206 INFO manager on server: list sched 2018-10-10 07:18:50,213 INFOCLI server: /opt/pbs/bin/qmgr -c list sched 2018-10-10 07:18:50,795 INFOCLI2 server: sudo -H /opt/pbs/sbin/pbsfs 2018-10-10 07:18:51,184 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2018-10-10 07:18:51,409 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2018-10-10 07:18:51,636 INFO manager on server: set server {'scheduler_iteration': '600', 'default_queue': 'workq'} 2018-10-10 07:18:51,646 INFOCLI server: /opt/pbs/bin/qmgr -c set server scheduler_iteration=600,default_queue=workq 2018-10-10 07:18:51,997 INFO status on server: resource 2018-10-10 07:18:52,006 INFO manager on server: list resource 2018-10-10 07:18:52,013 INFOCLI server: /opt/pbs/bin/qmgr -c list resource 2018-10-10 07:18:52,317 INFOCLI status on server: server license_count 2018-10-10 07:18:52,326 INFOCLI server: /opt/pbs/bin/qstat -Bf server.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:18:52,625 INFO server: server.ib0.smc-default.chf.rdlabs.hpecorp.net licensed 2018-10-10 07:18:53,161 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2018-10-10 07:18:53,918 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2018-10-10 07:18:54,142 INFO scheduler server: reverting configuration to defaults 2018-10-10 07:18:54,152 INFO manager on server: unset sched ['sched_priv', 'sched_cycle_length', 'scheduler_iteration', 'scheduling', 'sched_log'] 2018-10-10 07:18:54,159 INFOCLI server: /opt/pbs/bin/qmgr -c unset sched sched_priv,sched_cycle_length,scheduler_iteration,scheduling,sched_log 2018-10-10 07:18:54,528 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2018-10-10 07:18:54,750 INFOCLI2 server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2018-10-10 07:18:54,968 INFO scheduler server: reverting holidays file to default 2018-10-10 07:18:54,978 INFOCLI2 server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2018-10-10 07:18:55,204 INFOCLI2 server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2018-10-10 07:18:55,439 INFO scheduler server: sent signal -HUP 2018-10-10 07:18:55,451 INFOCLI2 server: sudo -H kill -HUP 4538 2018-10-10 07:18:55,681 INFOCLI2 server: sudo -H /opt/pbs/sbin/pbsfs -e -I default 2018-10-10 07:18:56,846 INFOCLI2 server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2018-10-10 07:19:03,845 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net /usr/bin/python -c "import sys; print sys.platform" 2018-10-10 07:19:11,784 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2018-10-10 07:19:15,412 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /opt/pbs/sbin/pbs_mom --version 2018-10-10 07:19:19,183 INFO mom mom@/etc/pbs.conf: reverting configuration to defaults 2018-10-10 07:19:19,194 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net which rm 2018-10-10 07:19:22,561 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2018-10-10 07:19:26,210 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2018-10-10 07:19:29,846 INFOCLI mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /opt/pbs/sbin/pbs_mom -s list 2018-10-10 07:19:58,875 INFOCLI mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /opt/pbs/sbin/pbs_mom -s list 2018-10-10 07:20:28,075 INFOCLI mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /opt/pbs/sbin/pbs_mom -s remove pbs_vnode_1539083155.def 2018-10-10 07:20:57,177 INFO status on server: host mom.ib0.smc-default.chf.rdlabs.hpecorp.net {'resources_available.host': 'mom.ib0.smc-default.chf.rdlabs.hpecorp.net', 'resources_available.vnode': None} 2018-10-10 07:20:57,189 INFOCLI server: /opt/pbs/bin/pbsnodes -s server.ib0.smc-default.chf.rdlabs.hpecorp.net -H mom.ib0.smc-default.chf.rdlabs.hpecorp.net 2018-10-10 07:20:57,496 ERROR err: ['Node: mom.ib0.smc-default.chf.rdlabs.hpecorp.net, Error: Unknown node '] 2018-10-10 07:20:57,508 INFO status on server: host mom {'resources_available.host': 'mom.ib0.smc-default.chf.rdlabs.hpecorp.net', 'resources_available.vnode': None} 2018-10-10 07:20:57,515 INFOCLI server: /opt/pbs/bin/pbsnodes -s server.ib0.smc-default.chf.rdlabs.hpecorp.net -H mom 2018-10-10 07:20:57,835 INFO manager on server: delete node mom[0],mom[1],vnode[0] 2018-10-10 07:20:57,845 INFOCLI server: /opt/pbs/bin/qmgr -c delete node mom[0],mom[1],vnode[0] 2018-10-10 07:20:58,199 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net python -c "import tempfile;print tempfile.mkstemp('PtlPbstmpcopy')[1]" 2018-10-10 07:21:02,373 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net which scp 2018-10-10 07:21:05,880 INFOCLI2 server: /usr/bin/scp /tmp/PtlPbsv8CI8P mom.ib0.smc-default.chf.rdlabs.hpecorp.net:/tmp/tmp7uKt_JPtlPbstmpcopy 2018-10-10 07:21:09,402 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net which cp 2018-10-10 07:21:12,811 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /usr/bin/cp /tmp/tmp7uKt_JPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2018-10-10 07:21:16,467 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net /usr/bin/rm /tmp/tmp7uKt_JPtlPbstmpcopy 2018-10-10 07:21:23,883 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2018-10-10 07:21:30,914 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2018-10-10 07:21:34,588 INFO running init script to stop pbs mom on mom.ib0.smc-default.chf.rdlabs.hpecorp.net using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2018-10-10 07:21:34,607 INFOCLI2 server: /usr/bin/scp -p /tmp/PtlPbshQbWpl mom.ib0.smc-default.chf.rdlabs.hpecorp.net:/tmp/PtlPbshQbWpl 2018-10-10 07:21:38,133 INFOCLI2 mom: ssh mom.ib0.smc-default.chf.rdlabs.hpecorp.net /tmp/PtlPbshQbWpl Contents of /tmp/PtlPbshQbWpl: ---------------------------------------- #!/bin/bash sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2018-10-10 07:21:43,012 INFO TIMEDOUT 2018-10-10 07:21:43,050 INFO 2018-10-10 07:21:43,056 INFO ====================================================================== 2018-10-10 07:21:43,061 INFO TIMEDOUT: test_sched_badstate_subjob (tests.functional.pbs_sched_subjob_badstate.TestSchedSubjobBadstate) 2018-10-10 07:21:43,067 INFO ___m_oo_m___ 2018-10-10 07:21:43,073 INFO Traceback (most recent call last): File "/tmp/ptl/lib/python2.7/site-packages/ptl/utils/pbs_testsuite.py", line 444, in setUp self.revert_moms() File "/tmp/ptl/lib/python2.7/site-packages/ptl/utils/pbs_testsuite.py", line 859, in revert_moms self.revert_mom(mom, force) File "/tmp/ptl/lib/python2.7/site-packages/ptl/utils/pbs_testsuite.py", line 940, in revert_mom rv = mom.revert_to_defaults(delvnodedefs=self.del_vnodes) File "/tmp/ptl/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 12784, in revert_to_defaults self.restart() File "/tmp/ptl/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 12636, in restart if not self.stop(): File "/tmp/ptl/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 12625, in stop self.pi.stop_mom() File "/tmp/ptl/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 14087, in stop_mom daemon='mom') File "/tmp/ptl/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 13934, in initd return self._unix_initd(hostname, op, conf_file, init_script, daemon) File "/tmp/ptl/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 14240, in _unix_initd logerr=False) File "/tmp/ptl/lib/python2.7/site-packages/ptl/utils/pbs_dshutils.py", line 951, in run_cmd (o, e) = p.communicate(input) File "/opt/pbs/python/lib/python2.7/subprocess.py", line 479, in communicate return self._communicate(input) File "/opt/pbs/python/lib/python2.7/subprocess.py", line 1098, in _communicate stdout, stderr = self._communicate_with_poll(input) File "/opt/pbs/python/lib/python2.7/subprocess.py", line 1152, in _communicate_with_poll ready = poller.poll() File "/tmp/ptl/lib/python2.7/site-packages/ptl/utils/plugins/ptl_test_runner.py", line 602, in timeout_handler raise TimeOut('Timed out after %s second' % timeout) TimeOut: Timed out after 180 second 2018-10-10 07:21:43,082 INFO ================================================================================ timedout: test_sched_badstate_subjob (tests.functional.pbs_sched_subjob_badstate.TestSchedSubjobBadstate) Test cases with failures: TestSchedSubjobBadstate.test_sched_badstate_subjob Test suites with failures: TestSchedSubjobBadstate run: 1, succeeded: 0, failed: 0, errors: 0, skipped: 0, timedout: 1 Tests run in 0:03:14.707022 2018-10-10 07:21:43,089 INFO Cleaning up temporary files 2018-10-10 07:21:43,098 INFO Cleaning up /var/tmp dir 2018-10-10 07:21:43,107 INFO Cleaning up /tmp dir