Improving the PTL Framework

Owned by Anne Urban
Last updated: Jul 03, 2019

This page appears in the PTL Developer's Guide.

Process for Improving PTL Framework

The process for improving the PTL framework is the same as the process you use for changing source code in PBS Professional.

PTL Directory Structure

Contents of pbspro/test/fw/:


DirectoryDescription of Contents
fw




bin

Test command: pbs_benchpress

PTL-based commands:

pbs_config: for configuring PBS and PTL for testing
pbs_loganalyzer: for analyzing daemon and accounting logs
pbs_snapshot: for capturing state of complex
pbs_stat: for getting status of PBS objects, e.g. jobs, queues

Commands that are not currently in use:

pbs_as: for future API mode
pbs_cov: for getting code coverage
pbs_py_spawn: left over from another project
pbs_swigify: for future API mode


doc

Documentation source .rst files

ptl

PTL package


lib

Core Python library: PBSTestLib

Supporting files for PBSTestLib: pbs_api_to_cli.py, pbs_ifl_mock.py



utils

PBS test suite: pbs_testsuite.py containing PBSTestSuite (which is Python testing framework)

Utilities PTL provides, such as pbs_snaputils.py, pbs_logutils.py, pbs_dshutils.py, pbs_crayutils.py, etc.




pluginsNose plugins for PTL framework

Contents of pbspro/test/tests/:


DirectoryDescription of Contents
tests


functional

Feature-specific tests

Test suites under this directory should inherit base class TestFunctional


interfaces

Tests related to PBS interfaces (IFL, TM, RM)

Test suites under this directory should inherit base class TestInterfaces


performance

Performance tests

Test suites under this directory should inherit base class TestPerformance


resilience

Server & comm failover tests

Stress, load, and endurance tests

Test suites under this directory should inherit base class TestResilience


security

Security tests

Test suites under this directory should inherit base class TestSecurity


selftest

Testing PTL itself

Test suites under this directory should inherit base class TestSelf


upgrades

Upgrade-related tests

Test suites under this directory should inherit base class TestUpgrades

Construction of PTL

PTL is derived from Python's unittest.

PTL uses nose plugins.

Testing PTL

Tests that test PTL are in pbspro/test/test/selftest, at https://github.com/PBSPro/pbspro/tree/master/test/tests/selftest

Enhancing PTL for New PBS Attributes

How to Add a New Attribute to the Library

This section is for PBS developers who may be adding a new job, queue, server, or vnode attribute, and need to write tests that depend on their new attribute. PTL does not automatically generate mappings from API to CLI, so when adding new attributes, it is the responsibility of the test writer to define the attribute conversion in ptl/lib/pbs_api_to_cli.py. They must also define the new attribute in ptl/lib/pbs_ifl_mock.py so that the attribute name can be dereferenced if the SWIG wrapping was not performed.
For example, let's assume we are introducing a new job attribute called ATTR_geometry that maps to the string "job_geometry".  In order to be able to set this attribute for a job:

  • We need to define it in pbs_api_to_cli.py as follows:

ATTR_geometry: "W job_geometry="

  • Add it to ptl/lib/pbs_ifl_mock.py as follows:

ATTR_geometry: "job_geometry"

  • In order to get the API to use the new attribute, rerun pbs_swigify, so that symbols from pbs_ifl.h are read in

Nose Plugins for PTL

PTL nose plugins are written for test collection, selection, observation, and reporting. Below are the various PTL plugins:

  • PTLTestLoader: Load test cases from a given set of parameters

  • PTLTestRunner: Run tests

  • PTLTestInfo: Generate test suite and test case information

  • PTLTestDb: Update the PTL execution data into a given data base type

  • PTLTestData: Save post-analysis data on test case failure or error

  • PTLTestTags: Load test cases from a given set of parameters for tag names

Nose supports plugins for test collection, selection, observation, and reporting.

Plugins need to implement certain mandatory methods. Plugin (hooks) fall into four broad categories: selecting and loading tests, handling errors raised by tests, preparing objects used in the testing process, and watching and reporting on test results.

Some of the PTL plugins, with their purposes, are listed below:

PTLTestLoader

Load test cases from given parameters.

Based on the parameters provided while running.  This plugin is necessary to load the required list of tests. It also checks for unknown test suites and test cases if any, and excludes any tests specified for exclusion.

PTLTestRunner

PTL plugin to run tests.

This plugin defines the following functionalities related to running tests:

  • Start or stop the test

  • Add error, success, or failure

  • Track test timeout

  • If post analysis is required, raise an exception when a test case failure threshold is reached

  • Log handler for capturing logs printed by the test case via logging module

  • Test result handling

PTLTestInfo

Load test cases from given parameters to get their information docstrings.

This plugin is used to generate test suite info, rather than running tests. This info includes test suite and test case docstrings in hierarchical order.  It can be queried using test suite names or tags.

PTLTestDb

Update the PTL execution data into a given data base type such as File, HTML, etc.

This plugin is for uploading PTL data into a data base.  PTL data includes all the information regarding a PTL execution such as test name, test suite name, PBS version tested on, hostname where the test is run, test start time, end time, duration, and test status (skipped, timed out, errored, failed, succeeded, etc.)  PTL data can be uploaded to any type of database, including File, JSON, HTML, and PostgresSQL.  The PTLTestDb plugin also defines a method to send analyzed PBS daemon log information either to the screen or to a database file, which is used with the pbs_loganalyzer test command.

PTLTestData

Save post analysis data on test case failure or error.

This plugin is for saving post analysis data on test case failure in PTL, along with tracking failures and handling test execution when failure thresholds are reached.

PTLTestTags

Load test cases from given parameters for tag names.

This plugin provides test tagging. It defines a Decorator function that adds tags to classes, functions, or methods, and loads the matching set of tests according to a given set of parameters.

Nose Documentation

http://nose.readthedocs.io/en/latest/testing.html
http://pythontesting.net/framework/nose/nose-introduction/
http://nose.readthedocs.io/en/latest/plugins/writing.html

Related Links and References

Complete Doxygenated PTL Documentation

http://www.pbspro.org/ptldocs/

Repository

https://github.com/PBSPro/pbspro
https://github.com/PBSPro/pbspro/test

Nose Documentation

http://nose.readthedocs.io/en/latest/testing.html
http://pythontesting.net/framework/nose/nose-introduction/
http://nose.readthedocs.io/en/latest/plugins/writing.html

Python Documentation

https://docs.python.org/2.7/tutorial/


OSS Site Map

Developer Guide Pages