Objective:

After PTL rpm Installation, the user has to cd to tests dir to execute test cases. We have -f option in pbs_benchpress but it supports only file. But we cannot use it to run a whole bunch of different tests.

when user runs pbs_benchpress -t testcase, it should do a path search – search current directory, next search test case in installed location.

Description

Currently, PTL doesn't support running test case from non-test directory using pbs_benchpress TestExecution option ('-t') i.e. a directory where all tests exist.

Though one can run by specifying the full path of the test file using pbs_benchpress FILE option ('-f').

This design document is to support pbs_benchpress to execute test cases(or test suite by its class name) from any directory on the host apart from tests directory.


Forum discussion at : http://community.pbspro.org/t/enhancement-of-pbs-benchpress-to-run-from-non-tests-directory/1660

Details:

File : utils/plugins/ptl_test_loader.py

Class : PTLTestLoader

Interface : PTL_TESTS_DIR

Detailed explanation of how PTL searches for a test case when executed using Execution option (pbs_benchpress -t):

  1. When a test case is run using "pbs_benhcpress Execution option -t" option, PTL will look for the test case in current directory, if it doesn't find then it will search in PTL installed directory (Ex: /opt/ptl/tests)
  2. User can also specify path to tests directory it is not present in PTL installed directory. If user wish to place tests in some non-default directory, then user can mention using environment variable "PTL_TESTS_DIR".                Ex: export PTL_TESTS_DIR=/home/user/temp/tests"
  3. If PTL_TESTS_DIR is mentioned, then PTL will search in PTL_TESTS_DIR directory only when it doesn't find test case in current directory.
  4. Currently PTL_TESTS_DIR is limited to single path specification. We can mention PTL_TESTS_DIR to point to single path.

Currently PTL supports getting test info only from tests directory. Now, one can get test info using pbs_benchpress INFO option (pbs_benchpress -i TestSuitName) from any directory apart from test directory.

Summary of order how a test case being searched:

1) current dir
2) user set dir(PTL_TESTS_DIR) (if set)
3) Installed dir (/opt/ptl/)

Example:

Assuming PTL is installed in "/opt/ptl/" path and tests directory is present under "/opt/ptl/tests path".

Scenario-1:

Consider running a single test case using pbs_benchpress -t option from /home/user/ directory (non-test directory) as pbsroot. 

Since, pbs_benchpress is run from /home/user/ path, it tries to find test case in current directory, when it fails to find then it will search in "/opt/ptl/tests/" and runs the test case successfully as it is present in "/opt/ptl/tests/".

Scenario-2:

Assuming tests directory is moved to "/tmp/tests/" path.

User need to export "PTL_TESTS_DIR" environment variable to above mentioned path.

Consider running a single test case using pbs_benchpress -t option from /home/user/ directory as pbsroot.

PTL first tries to find test case in current directory, if it fails to find there then it will search in user defined path given in environment variable "PTL_TESTS_DIR".

Since, test case is present in path mentioned in "PTL_TESTS_DIR", test case runs successfully.

Scenario-3:

Assuming tests directory is moved to "/tmp/tests/" path and user has not set PTL_TESTS_DIR env variable.

In this case when a test case is executed from /home/user directory, PTL throws error saying "Unknown test case" as it fails to find the test case in current directory as well as in PTL installed directory which is "/opt/ptl/tests/".

Scenario-4:

Assuming tests directory is moved to "/tmp/tests/" path and if user has set multiple paths in PTL_TESTS_DIR, then PTL will throw an error saying "Invalid directory specified in PTL_TESTS_DIR"

Ex: export PTL_TESTS_DIR=/tmp/tests:/home/user/tests