Description
Currently, PTL doesn't support running test case from non-test dircetory using pbs-benchpress TestExecution option ('-t') i.e. a directory where all tests exist.
Though one can run by specifying the full path of the test file using pbs_benchpress FILE option ('-f').
This design document is to support pbs_benchpress to execute test cases(or testsuite by it's class name) from any directory on the host apart from tests directory.
Forum discussion at : http://community.pbspro.org/t/enhancement-of-pbs-benchpress-to-run-from-non-tests-directory/1660
Details:
File : utils/plugins/ptl_test_loader.py
Class : PTLTestLoader
Interface : PTLTESTS
Detailed explainaion of how PTL searches for a test case when executed using Execution option (pbs_benchpress -t):
- When a test case is run using "pbs_benhcpress Execution option -t" option, PTL will look for the test case in current dircetory, if it doesn't find then it will search in PTL installed dircetory (Ex: /opt/ptl/tests)
- User can also specify path to tests dircetory it is not present in PTL installed directory. If user wish to place tests in some non-default dircetory then user can mention using environment variable "PTLTESTS". Ex: export PTLTESTS=/home/user/temp/tests"
- If PTLTESTS is mentioned then PTL will serach in PTLTESTS dircetory only when it doesn't find test case in current and PTL installed directory.
- Currently PTLTESTS is limited to single path specification. We can mention PTLTESTS to point to single path.
Currently PTL supports getting test info only from tests directory. Now, one can get test info using pbs_benchpress INFO option (pbs_benchpress -i TestSuitName) from any dircetory apart from test dircetory.
Example:
Assuming PTL is installed in "/opt/ptl/" path and tests dircetory is present under "/opt/ptl/tests path".
Scenario-1:
Consider running a single test case using pbs_benchpress -t option from /home/user/ dircetory(non-test dircetory) as pbsroot.
- pbs_benchpress -t SmokeTest.test_submit_job
Since, pbs_benchpress is run from /home/user/ path, it tries to find test case in current dircetory, when it fails to find then it will search in "/opt/ptl/tests/" and runs the test case successfully as it is present in "/opt/ptl/tests/".
Scenario-2:
Assuming tests dircetory is moved to "/tmp/tests/" path.
User need to export "PTLTESTS" environment variable to above mentioned path.
- export PTLTESTS=/tmp/tests
Consider running a single test case using pbs_benchpress -t option from /home/user/ dircetory as pbsroot.
- pbs_benchpress -t SmokeTest.test_submit_job
PTL first tries to find test case in current dircetory, when it fails to find then it will search in "/opt/ptl/tests/" path. If it fails to find there as well then it will search in user defined path given in environment variable "PTLTESTS".
Since, test case is present in path mnetioned in "PTLTESTS", test case runs successfully.
Scenario-3:
Assuming tests dircetory is moved to "/tmp/tests/" path and user has not set PTLTESTS env variable.
In this case when a test case is executed from /home/user directory, PTL throws error saying "Unknown testcase" as it fails to find the test case in current dircetory as well as in PTL installed directory which is "/opt/ptl/tests/"
Project Documentation Main Page