Description
Currently, PTL doesn't support running test case from non-test directory using pbs-benchpress TestExecution option ('-t') i.e. a directory where all tests exist.
Though one can run by specifying the full path of the test file using pbs_benchpress FILE option ('-f').
This design document is to support pbs_benchpress to execute test cases(or test suite by it's class name) from any directory on the host apart from tests directory.
Forum discussion at : http://community.pbspro.org/t/enhancement-of-pbs-benchpress-to-run-from-non-tests-directory/1660
Details:
File : utils/plugins/ptl_test_loader.py
Class : PTLTestLoader
Interface : PTL_TESTS_DIR
Detailed explanation of how PTL searches for a test case when executed using Execution option (pbs_benchpress -t):
Currently PTL supports getting test info only from tests directory. Now, one can get test info using pbs_benchpress INFO option (pbs_benchpress -i TestSuitName) from any directory apart from test directory.
Example:
Assuming PTL is installed in "/opt/ptl/" path and tests directory is present under "/opt/ptl/tests path".
Scenario-1:
Consider running a single test case using pbs_benchpress -t option from /home/user/ directory(non-test directory) as pbsroot.
Since, pbs_benchpress is run from /home/user/ path, it tries to find test case in current directory, when it fails to find then it will search in "/opt/ptl/tests/" and runs the test case successfully as it is present in "/opt/ptl/tests/".
Scenario-2:
Assuming tests directory is moved to "/tmp/tests/" path.
User need to export "PTL_TESTS_DIR" environment variable to above mentioned path.
Consider running a single test case using pbs_benchpress -t option from /home/user/ directory as pbsroot.
PTL first tries to find test case in current directory, when it fails to find then it will search in "/opt/ptl/tests/" path. If it fails to find there as well then it will search in user defined path given in environment variable "PTL_TESTS_DIR".
Since, test case is present in path mentioned in "PTL_TESTS_DIR", test case runs successfully.
Scenario-3:
Assuming tests directory is moved to "/tmp/tests/" path and user has not set PTL_TESTS_DIR env variable.
In this case when a test case is executed from /home/user directory, PTL throws error saying "Unknown testcase" as it fails to find the test case in current directory as well as in PTL installed directory which is "/opt/ptl/tests/"