Design for a supported way to change default setup in PTL
Currently in PTL it is not possible to test an existing test case with a newly introduced feature configuration in PBS, since PTL's setUp() method would revert the newly enabled feature to Out Of Box settings (default values at PBS' fresh installation setup). In order to do this we need to rewrite the test with this new configuration enabling coded in the test. Another case is a specific set of values of existing PBS configurations to which one would want to revert to every time before running a test case; which is also not possible right now. Example, I would always want to retain a queue over the span of whole test suite run. This leads to necessity of a supported way of describing a given setup to PTL and make PTL use that setup as default setup before running any test case.
Forum Discussion: http://community.pbspro.org/t/design-for-a-supported-way-to-change-default-setup-in-ptl/1096
External changes to PTL:
1. New pbs_benchpress command option: "--use-current-setup":
With this option, PTL tests will be executed on a set of altered PBS configurations which are on the current test set up; with tests not being run after reverting PBS to Out of Box setup.
Without this option, PTL would revert PBS to Out of box configurations and then run the tests on it; which is the case right now.
2. At the end of test run, PTL will now revert PBS to the same set of configurations on which it started its tests:
For example if the tests are started on Out of Box PBS settings, PTL will revert PBS to Out of Box at the end of test run. Right now PTL does not do the same at the end.
Otherwise, if PTL tests are started on an altered set of PBS configurations that differ from Out of Box; then, the same configurations are saved and at the end of the test run PTL would revert PBS to this saved set of PBS configurations.
Thus PBS configurations remain the same before and after PTL test run.
Details of updates to the PTL framework:
A. pbs_benchpress option: --use-current-setup
Interface: --use-current-setup
Visibility: Public
Change Control: Stable
Details:
A new option called “--use-current-setup” is added to pbs_benchpress to make PTL use current PBS configurations on the test machine instead of the “out-of-box” configurations; while running a test. This is done by updating the PBSTestSuite methods setUpClass(), setUp(), tearDown() & tearDownClass(). PTL will utilize the existing "save_configuration()" and "load_configuration()" methods (along with their enhancements) to save and reload the set of PBS configurations.
When pbs_benchpress option "--use-current-setup" is present, the tests are run on the PBS configurations saved before starting the tests. When pbs_benchpress option "--use-current-setup" is absent, the tests are run on out-of-box PBS configurations.
A.1 pbs_benchpress retains PBS configurations that existed before running test:
pbs_benchpress execution will now save the PBS configurations on which it started; runs the tests and then reverts PBS back to the same state on which it started. This behavior is independent of presence or absence of option "--use-current-setup".
B. PBSTestSuite setUpClass() & tearDownClass() method updates:
PBSTestSuite's setUpClass() method will save the current PBS configurations by calling save_configuration() method. When pbs_benchpress is executed with option 'use-current-setup', the current PBS configurations are saved by PBSTestSuitesetUpClass(). Before running the tests, there are certain users i.e. test execution user and MGR_USER(PTL specific) added as manager role and OPER_USER as operator role in PBS server. The objective of adding the users is to make sure the behaviour of PTL remains same with or without --use-current-setup. The tests are run on the same current set of PBS configurations without reverting PBS to out of the box. When the option 'use-current-setup' is absent, the current PBS configurations are saved; then PBSTestSuitesetUpClass() will revert the current PBS test setup to out of the box by calling all the daemons' revert_to_defaults() methods, instead of them being called at PBSTestSuite's setUp() and then the tests are run on it.
Feature test suite specific configurations that need to be maintained all through the test run need to be coded in custom setUpClass() with call to PBSTestSuite setUpClass() call before to it. i.e. set up some things once and leave them as is during the whole test run.
PBSTestSuite tearDownClass() will now revert PBS configurations to either out of the box or the saved configuration on which it started tests in case of PBS under test differing from out of box configurations.
C. PBSTestSuite setUp() & tearDown() method updates:
PBSTestSuite setUp() method now saves the current PBS configurations on the test set up where the test is run; by calling save_configuration() method. This saved data is used by load_configuration() method to get the test set up PBS configurations to a given state. This saving happens only when the first time the PBSTestSuite setUp() method is called.
PBSTestSuite tearDown() method loads the configuration saved in setUp(). No action is taken when the setUp() method is called during the run second or further test cases in the same pbs_benchpress execution unless the load_configuration() fails in tearDown(). A switch called PBSTestSuite.config_saved will be used to control this behavior of setUp(). The switch is by default False, once the setUp() saves the configuration, it sets this switch to True.
save_configuration() will save the following from the PBS cluster:
Commands output:
qmgr print server
qmgr print sched
qmgr print hook
qmgr export hook <each hook> application/x-python default
qmgr export hook <each hook> application/x-config default
qmgr print resource
qmgr list pbshook
qmgr export pbshook <each pbshook> application/x-config default
pbsnodes
Configuration files:
sched_priv/sched_config
sched_priv/holidays
sched_priv/resource_group
sched_priv/dedicated_time
multiple schedulers directories if any
mom_priv/config (files of each hosts listed in pbsnodes)
mom_priv/config.d/ (saves vnode definition files listed with "pbs_mom -s list")
/etc/pbs.conf (all nodes)
load_configuration() will load these saved configurations if altered on the PBS cluster
Explanation:
Details of command line - PBS test set up combinations and their effect on setUp() & tearDownClass() are as follows:
| PTL test run command | Test setup on node | PBSTestSuite setUpClass() | Test suite specific setUpClass() | PBSTestSuite setUp() action | Test suite specific setUp() | PBSTestSuite tearDown() action | PBSTestSuite tearDownClass() | |
|---|---|---|---|---|---|---|---|---|
| 1 | pbs_benchpress | Out Of Box PBS - S0 | -reverts PBS to OOB - S0 | - | -Saves current PBS - S0 -loads the saved PBS if failed in tearDown | - | -loads the saved PBS S0 everytime | -reverts PBS to OOB - S0 |
| 2 | pbs_benchpress | Out Of Box PBS- S0 | -reverts PBS to OOB - S0 | -Applies test suite specific configurations S1 over S0 | -Saves current PBS S1 for first time -loads the saved PBS if failed in tearDown | -Applies test suite specific configurations S2 everytime | -loads the saved PBS S1 everytime | -reverts PBS to OOB - S0 |
| 3 | pbs_benchpress | User updated PBS - SA | -reverts PBS to OOB - S0 -Saves current PBS - SA | - | -Saves current PBS - S0 -loads the saved PBS if failed in tearDown | - | -loads the saved PBS S0 everytime | -reverts PBS to SA |
| 4 | pbs_benchpress | User updated PBS - SA | -reverts PBS to OOB - S0 -Saves current PBS - SA | -Applies test suite specific configurations S1 over S0 | -Saves current PBS - S1 -loads the saved PBS if failed in tearDown | -Applies test suite specific configurations S2 everytime | -loads the saved PBS S1 everytime | -reverts PBS to SA |
| 5 | pbs_benchpress --use-current-setup | User updated PBS - SA | -Saves current PBS - SA | - | -Saves current PBS - SA -loads the saved PBS if failed in tearDown | - | -loads the saved PBS SA everytime | -reverts PBS to SA |
| 6 | pbs_benchpress --use-current-setup | User updated PBS - SA | -Saves current PBS - SA | -Applies test suite specific configurations with PBS now in SA1 | -Saves current PBS - SA1 -loads the saved PBS if failed in tearDown | -Applies test suite specific configurations with PBS moving to state SA2 everytime | -loads the saved PBS SA1 everytime | -reverts PBS to SA |
| 7 | pbs_benchpress --use-current-setup | Out Of Box PBS - S0 | -Saves current PBS - S0 | - | -Saves current PBS - S0 -loads the saved PBS if failed in tearDown | - | -loads the saved PBS S0 everytime | -reverts PBS to OOB - S0 |
| 8 | pbs_benchpress --use-current-setup | Out Of Box PBS - S0 | -Saves current PBS - S0 | -Applies test suite specific configurations with PBS now in S1 | -Saves current PBS - S1 -loads the saved PBS if failed in tearDown | -Applies test suite specific configurations with PBS moving to state S2 everytime | -loads the saved PBS S1 everytime | -reverts PBS to OOB - S0 |
Below is a flow chart that explains the PBS states and the PTL tests flow with and without "--use-current-setup" option. (1 to 6 of above table)
Note:
1. Option use-current-setup applies to the whole pbs_benchpress run and not to a single or subset of tests of a test run.
2. If no PBS configurations are altered on a test set up; then there is no effect of "use-current-setup" option on the test run.
3. load_configuration() will take care of setting the configuration back if not set, i.e. if res1 is not present create it, else if res1 is present do not create it.
4. Ideally a test that needs mandatory configurations should have a check for the same at start and should not continue if it is absent.
5. The test_execution user, MGR_USER and OPER_USER added during the setupclass does not get reverted.