Using pbs_benchpress
- Anne Urban
This page appears in the PTL Developer's Guide.
You use pbs_benchpress to run or list test suites and test cases.
How to run pbs_benchpress
Go to pbspro/test/tests in your cloned GitHub directory structure.
Syntax:
pbs_benchpress <option(s)
Privilege
By default, pbs_benchpress automatically makes its current user a PBS Manager, via "qmgr -c 's s managers += < current user>".
Options to pbs_benchpress
--cumulative-tc-failure-threshold=<count>
Cumulative test case failure threshold. Defaults to 100.
Set count to 0 to disable this threshold.
If not zero, must be greater than or equal to 'tc-failure-threshold'
--db-access=<path to DB parameter keyword file>
Path to a file that defines DB parameters (PostreSQL only); can be absolute or relative.
See https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS.
You probably need to specify the following parameters:
dbname: database name
user: user name used to authenticate
password: password used to authenticate
host: database host address
port: connection port number
--db-name=<database name>
Database where test information and results metrics are stored for later processing.
Defaults to ptl_test_results.db.
--db-type=<database type>
Can be one of "file", "html", "json", "pgsql", "sqlite".
Defaults to "json".
--eval-tags='<Python expression>'
Selects only tests where Python expression evaluates to True. Can be applied multiple times.
Example:
'smoke and (not regression)' - Selects all tests which have "smoke" tag and don't have "regression" tag
'priority>4' - Selects all tests which have "priority" tag and where the value of priority is greater than 4
--exclude=<tests to exclude>
Comma-separated string of tests to exclude
-f <test files to run>
Comma-separated list of test file names
-F
Sets logging format to include timestamp and level
--follow-child
If set, walks the test hierarchy and runs each test. Starts from current directory.
-g <path to test suite file>
Absolute path to file containing comma-separated list of test suites
Used for very long list of tests
--genhtml-bin=<path to genhtml binary>
Defaults to "genhtml": PTL assumes genhtml is in PATH
-h
Displays usage information
-i
Prints test name and docstring to stdout, then exits.
-l <log level>
Specifies log level to use. Log levels are documented in PTL.
-L
Prints the following to stdout, then exits: test suite, file, module, suite docstring, tests, and number of test suites and test cases in current directory and all its subdirectories.
--lcov-baseurl=<url>
Base URL for lcov HTML report. Uses url as base URL.
--lcov-bin=<path to lcov binary>
Defaults to "lcov": PTL assumes lcov is in PATH
--lcov-data=<path to directory>
Path to directory containing .gcno files; can be absolute or relative
--lcov-nosrc
If set, do not include PBS source in coverage analysis.
By default, PBS source is included in coverage analysis
--lcov-out=<path to output directory>
Output path for HTML report.
Default path is TMPDIR/pbscov-YYYYMMDD_HHmmSS
--list-tags
Recursively list all tags in tests in current directory and its subdirectories.
--log-conf=<path to logging config file>
Absolute path to logging configuration file. No default.
File format specified in https://docs.python.org/2/library/logging.config.html#configuration-file-format.
--max-postdata-threshold=<count>
Max number of diagnostic tool runs per testsuite. Diagnostic tool is run on failure. Defaults to 10 sets of output from diagnostic tool.
Set count to 0 to disable this threshold.
--max-pyver=<version>
Maximum Python version
--min-pyver=<version>
Minimum Python version
-o <logfile>
Name of log file.
Logging output is always printed stdout; this option specifies that it should be printed to specified log file as well.
-p <test parameter>
Comma-separated list of <key>=<value> pairs. Note that the comma cannot be used in value. Pre-defined pairs are in 'Custom parameters:' in http://pbspro.org/ptldocs/ptl.html#ptl.utils.pbs_testsuite.PBSTestSuite but you can add your own along with those.
--param-file=<parameter file>
If set, gets test parameters from file
Overrides -p
--post-analysis-data=<post-analysis data directory>
Path to directory where diagnostic tool output is placed.
--stop-on-failure
If set, stop when any test fails
-t <test suites>
Comma-separated list of test suites to run
--tags=<list of tags>
Selects only tests that have tag tag. Can be applied multiple times.
Format: [!]<tag>[,<tag>]
Examples:
smoke - Selects all tests which have "smoke" tag
!smoke - Selects all tests which don't have "smoke" tag
smoke,regression - Selects all tests which have both "smoke" and "regression" tag
--tags-info
Lists all selected test suites
When used with --verbose, lists all test cases
Must be used with --tags or --eval-tags
You can use -t or --exclude to limit selection
--tc-failure-threshold=<count>
Test case failure threshold per testsuite
Defaults to 10
To disable, set count to 0
--timeout=<seconds>
Duration after which no test suites are run
--verbose
Shows verbose output
Must be used with -i, -L or --tags-info
--version
Prints version number to stdout and exits