PTL FAQs and Tips & Tricks

Owned by Anne Urban
Last updated: Sept 04, 2018

FAQs

  1. When I run "pbs_benchpress -t TestCgroups", I get "unknown testsuite: TestCgroups" error, even though "TestCgroups" is a valid test suite name. How do I resolve this?

    1. First, try to run python -tt <ptl_test_file> to verify that no syntax errors were made
    2. If the test file has executable permission, remove it (recommended permission for test file is 0644)
    3. If that doesn't work, provide the path to the ptl_test_file using the -f option, e.g. pbs_benchpress -f pbs_cgroups_tests.py -t TestCgroups

  2. Should I run the PTL test suite on my production environment?

    1. Never run PTL test cases in the production environment. It will only take once for you to regret it. PTL will delete all the jobs and restore the system to the default configuration!

  3. Where should I place the new tests?

    1. Put them under pbspro/test/tests.  See our test directory structure.

  4. What should I name the tests?

    1. Follow our PTL naming conventions!

  5. How can I specify a file that pbs_benchpress can't find?

    1. pbs_benchpress -f <name of test file>  For example, pbs_benchpress -f my_tests.py

  6. How can I run only a specific test?

    1. Specify the test using the -t option.  For example, if you have a test suite called TestCgroups in a file named pbs_cgroups_tests.py, execute the following:

      1. pbs_bench press -f pbs_cgroups_tests.py -t TestCgroups

    2. To run two specific tests in the TestCgroups suite:

      1. pbs_benchpress -f pbs_cgroups_tests.py -t TestCgroups.test_t4b,TestCgroups.test_t4c

  7. Is custom tearDown() necessary? if we define custom tearDown() explicitly in the test, do we need to call parent class teardown()?

    1. If the test case is doing some operations which PTL does not revert, for example "changing system configuration", the test case should define custom tearDown().

    2. If custom tearDown() is defined in the test, it should perform test-case-specific reverting operations.  For example if it does "changing system configuration", it should call parent class tearDown().

    3. If the PTL revert function doesn’t revert some PBS settings, it should be treated as a PTL bug/enhancement.

  8. How can I run all tests under the 'functional' directory?

    1. Try the "--follow-child" option to pbs_benchpress:
      1. pbs_benchpress -t TestFunctional --follow-child

  9. Can I submit a PTL test which fails now, but will pass when condition x is met? (e.g., when a known bug is fixed, or a known PTL enhancement is implemented)

    1. Your GitHub pull request should use only those new/modified PTL tests which pass, as evidenced by test logs. If there's a test case that is important, but has a dependency on some other code change, you have a few options:
      1. If your code change is not time-critical, then the best approach is to just hold off until the dependency is resolved.
      2. If your code change is time-critical and you have an entire test case/test suite which will fail, then you could use PTL's skip functionality to skip it and explain the dependency, then add a new ticket which should be linked to the dependency so that it can be tracked.
      3. If there's only a small section of a test case which would cause failure, and it's an important check, then you could again create a ticket, put that section there, and link it to the dependency so that it can be added once it's resolved.


Tips and Tricks

  • If you want to write something in a PTL test and are not sure how, check the existing PTL tests.  One of them might do something similar to what you want to do.
  • When an assert fails, the values of what failed will be printed into the PTL log (e.g. ‘0 is not True’ or ‘5 is not equal to 3’).  The failure message you pass in should provide more information for why your test is failing.
  • When putting a long (15+ seconds) sleep into your test, be kind to your test runner and put a self.logger.info() message above it to say why you are sleeping.  For example, self.logger.info(‘Waiting 30s for reservation to start running’)
  • PTL will revert back to a default state between tests.  Your test doesn’t have to do this cleanup itself.
  • If there are common things to set up your environment or things to clean up in many tests, consider using the common setUp() or tearDown() functions.  They are called before/after each test.  If you do, make sure to call your parent class' setUp() (e.g. TestFunctional.setUp()) or tearDown().
    • If you are creating/modifying something outside of PBS (e.g. creating a file), clean it up in tearDown().  If you were to try and clean it up at the end of your test, it might not be cleaned up.  If your test fails, you won’t reach the end of the test case.
  • Tests are not run in the order they appear in the file.  Don’t write tests that are dependent on each other.


OSS Site Map

Developer Guide Pages