1. Contents

1.1 Abbreviations

API:   Application Programming Interface
CLI:   Command-line Interface
fw:     framework
PBS:  PBS Professional
PTL:  PbsTestLab

2. PTL Framework Components

2.1 Interfaces to PBS

PTL executes tests with PBS operations using interfaces including the following:

• PBS IFL-API
• PBS commands
• PBS configuration files
  • sched_config
  • mom_priv/config
  • the holidays file
• the PBS daemon log files

PTL provides two ways of running tests:

• IFL API mode (PTL_API)
• Command Line mode (PTL_CLI)

2.2 PBS Entities: Core Library

The PTL library provides PBS IFL operations through either SWIG wrappers or the PBS CLI.  It contains classes that encapsulate the following PBS entities:

• Server
• Scheduler
• MoM
• Comm
• Queue
• Job
• Reservation
• Hook
• Resource

The PTL libary includes utility classes to convert PBS data structures and attributes to Python lists, strings, and dictionaries. It provides APIs for high-level PBS operations to operate on PBS entities.  The PTL library provides the below PBS entity objects used in writing tests:

• Server Object: PBS server configuration and control
• Scheduler Object: Container of scheduler-related properties
• MoM Object: Container for MoM properties. Provides various MoM operations
• Comm Object: PBS comm configuration and control
• Job Object: PBS job attributes and resources
• Reservation Object: PBS reservation attributes and resources
• Queue Object: PBS Queue container.  Holds queue attributes and pointer to server
• Hook Object: PBS hook objects. Holds attribute information and pointer to server

2.3 Distributed Shell Utilities

The PBS shell utilities are a set of functions to run commands, copy files, get process information, and parse a PBS configuration on an arbitrary host.  Below are a few examples of the functions that these utilities provide:

• Get local or remote platform info
• Get the path of the pbs.conf file
• Set/unset environment or configuration variables in a pbs.conf file
• Run a command on a host or list of hosts
• Copy a file or directory to specified target hosts
• Wrapper method to run PTL command
• Generic functions include chmod, chown, chgrp, rm, mkdir, useradd, userdel, with remote host support, etc.

2.4 Log Utilities

Miscellaneous utilities to process and analyze PBS log files.  Contains utilities to parse and report metrics from server, scheduler, MoM, and accounting logs. Metrics include number of jobs queued, run, ended, cycle duration, and in-scheduler duration.  Below are a few examples of the utilities and functions:

• Get the number of lines of a daemon log
• Helper function to parse log file
• Function to match a line in log file
• Analyze server, scheduler, MoM, and accounting logs
• Get PBS job information
• Etc.

2.5 Testing Utilities

PTL provides testing utilities as modules that aid in building tests using the PTL objects of PBS entities.

These are in pbspro/test/fw/ptl/utils (https://github.com/PBSPro/pbspro/tree/master/test/fw/ptl/utils)

Below are a few of the testing utilities:

pbs_testsuite

This utility defines a main master test suite class called PBSTestSuite that defines generic setup, teardown, and logging functions to be used. It is taken as parent class for most tests. This class by default instantiates server, scheduler, MoM, and comm objects connected to localhost. It also provides a wrapper class to handle multiple service instances (i.e., MoM, server, and scheduler) and certain aiding functions such as a function to skip a test, a function to check modules available on the system, etc. This utility is explained in detail in a later section of this document.

pbs_cliutils

Generic system command interface utility that provides functions such as:

• Check for the existence of a binary (a command) using the "which" command
• Expand a relative path to an absolute path
• Return a list of privileged ports in use on a given host
• getting PTL logging levels
• Etc.

pbs_covutils

Code coverage utilities.  Below are a few of the functionalities provided by these utilities:

• Set the coverage binary
• Set the HTML generation utility
• Set the coverage output directory
• Set the coverage data directory
• Initialize coverage
• Capture the coverage parameters
• Summarize the coverage output
• Generate the HTML report
• Etc.

pbs_procutils

Utilities to query process information.  Below are a few of the utilities and functions provided:

• Get process information from a process name or PID on a given host
• Get PID's process state on host
• Get process information reports for PID such as RSS and VSZ, along with the respective times
• Background process monitoring tool
• Etc

pbs_fileutils

Utility to walk a file using the result of 'head' or 'tail' on the local file system. Below are a few of the functions provided:

• Return the file descriptor associated with the file being processed
• Get the size of the file
• Get the next n lines of the file from head or tail
• Get the nth line of the file
• Get a block of lines in a file
• Etc.

2.6 Test Commands, Log Analyzer, Stat, and Filter

This layer contains the following test commands:

• pbs_benchpress: Command used to run or list test suites and test cases
• pbs_loganalyzer: Analyze PBS logs and return various throughput metrics
• pbs_stat: Status and filter PBS objects based on select properties
• pbs_swigify: Build IFL swig wrappers and copy them over to the library
• pbs_as: Used by the library to impersonate a user for API job submission
• pbs_config: Configure PBS: configure services, create and configure vnodes, etc.
• pbs_py_spawn: Wrapper to the pbs_py_spawn API call
• pbs_cov: Generates lcov/ltp (gcov) coverage analysis. Requires instrumented source files

The link below explains how to use the above test commands:
https://pbspro.atlassian.net/wiki/display/DG/Using+PTL+for+Testing#UsingPTLforTesting-Overviewofcommands

3. PTL Library: Objects and APIs

The PTL core library consists of objects encapsulating PBS entities.  These entities are described in the following sections.

3.1 PBS Server Object

For PBS server configuration and control. 
A server object is an instance of the Server class.  A server object is a container for PBS server attributes, and uses wrappers for the IFL API to perform operations on the server.  For example, you can use it to submit, status, delete, manage, etc. jobs, reservations, and configurations.
The server class also offers higher-level routines to ease testing.  Examples: a function that reverts the PBS server configuration to prior known "out of the box" settings, a function to expect an attribute to match a given value as per an operation, and a function to return the count of anything queried in the PBS setup such as the  number of nodes etc. based on certain conditions.
The class contains routines to act upon the PBS server daemon, such as starting, stopping, or restarting the PBS server daemon, and a function to match log messages in the daemon log files.
The server class also contains a dictionary that holds general configuration for the framework's operations, such as operation mode (PTL_CLI or PTL_API), maximum attempts to try while checking for a command in a test condition, and the interval between which this check of the command happens.
Below is a depiction of the Server class hierarchy:

• Server: The Server class
• PBSService: Generic PBS service object to hold properties of PBS daemons
• PBSObject: Generic PBS object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.2 PBS Scheduler Object

Container of Scheduler-related properties. 
The Scheduler class contains routines to update scheduler configuration along with routines to act upon the PBS scheduler daemon, such as starting, stopping, or restarting the PBS scheduler daemon, and function to match log messages in the daemon log files.
This class also offers higher-level routines to ease testing.   Examples: a function that reverts the PBS scheduler configuration to prior known "out of the box" settings, and a function to save daemon configurations.
Below is a depiction of the Scheduler class hierarchy:

• Scheduler: The Scheduler Class
• PBSService: Generic PBS service object to hold properties of PBS daemons
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.3 PBS MoM Object

Container for MoM properties. 
The MoM class contains routines to update MoM attributes and act upon the PBS MoM daemon. Provides various MoM operations, such as creation, insertion, and deletion of vnodes.
This class also offers higher-level routines to ease testing.  Examples: a function that reverts the PBS MoM configuration to prior known "out of the box" settings, and a function to save daemon configuration.
Below is a depiction of the MoM class hierarchy:

• MoM: The MoM Class
• PBSService: Generic PBS service object to hold properties of PBS daemons
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.4 PBS Comm Object

For PBS Comm configuration and control. 
This class also offers higher-level routines to ease testing, for example: a function that reverts the PBS comm configuration to prior known "out of the box" settings, and a function to save daemon configurations.
Below is a depiction of the comm class hierarchy:

• Comm: The Comm Class
• PBSService: Generic PBS service object to hold properties of PBS daemons
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.5 PBS Job Object

Container for PBS Job attributes and resources. 
This class defines certain default options of a PBS job including job name, job directory, mail_point, job priority, rerunnable, option to keep job output-error files on execution node, etc. These are overridden when any exclusive specifications are made after job object instantiation. Below are a few methods defined inside this class:

• Set the sleep duration for this job
• Set the executable and arguments to use for this job
• Create a job script from a given body of text and put it in a temporary location
• Customize the ``Variable_List`` job attribute to ``<user>``

Below is depiction of the job class hierarchy:

• Job: The Job Class
• ResourceResv: Generic PBS resource reservation (a job or advance or standing reservation)
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.6 PBS Reservation Object

Container for PBS Reservation attributes and resources. It initializes default start and end time; overridden when exclusive specifications are made. 

Below is a depiction of the reservation class hierarchy:

• Reservation: The Reservation class
• ResourceResv: Generic PBS resource reservation (a job or advance or standing reservation)
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.7 PBS Queue Object

Container for a PBS queue.  Holds attributes of the queue and a pointer to the server. Also defines a method to revert queue attributes to defaults.
Below is a depiction of the queue class hierarchy:

• Queue: The Queue Class
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

3.8 PBS Hook Object

Represents a PBS hook.  Holds attribute information and a pointer to the server.
Below is a depiction of the class hierarchy:

• Hook: The Hook class
• PBSObject: Generic PBS Object encapsulating attributes and defaults
• Object: The most basic type in built-in Python classes (_builtin_.py)

4. Communication of PBS Data in PTL

A test step involves sending the input data necessary for PBS operations to the PTL API and returning of the results of this PBS operation in test-understandable format. In this flow, based on the PTL operation mode, the below data formats are involved:

• Python data structures: list, str, dict, tuple, list of dict, list of key=value pairs, etc.
• PBS data structures: attrl, attropl, batch_status objects, etc. (defined in PBS)
• PBS command argument mapping: mapped strings of PBS command-line arguments (defined in pbs_api_to_cli.py)

4.1 PBS Data Flow and Conversion in PTL

BatchUtils is a Utility class to create, convert, or display various PBS data structures. It is defined in PTL's core library (pbs_testlib) and is needed to convert the data as depicted below for PBS operations in PTL.

4.1.1 Data Flow and Conversion During Testing of PBS

Testing PBS Operation in API Mode

1. The test provides the necessary PBS data in Python data structures to PTL API
2. This data is converted to PBS data structure format using functions defined in BatchUtils
3. The data is then passed to PBS IFL Wrapper functions to execute the test step

Testing PBS Operation in CLI Mode

1. The test provides the necessary PBS data in Python data structures to PTL API
2. This data is mapped to PBS command arguments using functions defined in BatchUtils
3. Then the whole PBS command is run by PTL

4.1.2 Data Flow and Conversion for Result of Testing PBS

Testing Result of PBS Operation in API Mode

1. The result of a PBS operationin API mode returns in PBS data structure format. 
2. This result is converted to Python format using functions defined in BatchUtils.

Testing Result of PBS Operation in CLI Mode

1. The result of a PBS operation in CLI mode returns in the PBS format made up of a list of "key=value" pairs. 
2. This result is converted to convenient Python data structure format using functions defined in BatchUtils.

4.2 Flow of Methods During Test Case Execution

Below is the flow of methods inside a PTL test suite that is inherited from the master test suite, named PBSTestSuite, when test cases are executed:

5. PBSTestSuite

5.1 Overview of PBSTestSuite

PBSTestSuite is a master test suite class that defines generic setup, teardown, and logging functions to be used by inherited test suites. This class instantiates server, scheduler, MoM, and comm objects connected to localhost. This class has functions that include deletion of all jobs and reservations, reversion of server and scheduler configuration to defaults, and checks to ensure that there is at least one CPU to schedule work on before running a test. This class also accepts a bunch of custom parameters based on the necessary test setup.

5.2 Parent Methods Defined in the Parent Class (PBSTestSuite)

• Parent SetUpClass()

• Parent SetUp()

• Parent TearDown()

• Parent TearDownClass()

5.3 Custom Methods Defined In the Test Suite

• Custom SetUpClass()
• Custom SetUp()
• Custom TearDown()
• Custom TearDownClass()

5.4 Reverting to Defaults

Server, scheduler, MoM, and queue each have a method to revert to defaults.  Here is a description of what each one does:

5.4.1 Reverting Server to Defaults

• Delete jobs and reservations
• Unset all qmgr options that are different from out-of-the-box PBS options
• Delete all non-default queues that the test may have created. Options exist to not delete the queues and instead to disable non-default queues
• Delete all hooks that a test may have created. Options exist to instead disable hooks
• Option exists to delete resourcedef file. Default is to not delete resourcedef file
• Vnodes are not deleted from the server object; it is the responsibility of the test writer to delete/create vnodes as needed at the beginning of the test

5.4.2 Reverting Queue to Defaults

• Unset queue attributes that are different from out-of-the-box queue attributes

5.4.3 Reverting Scheduler to Defaults

• Clear any dedicated time that may have been set
• Reset sched_config to out-of-the-box defaults
• Send a HUP signal to the scheduler so that it re-reads its configuration file
• Unset scheduler attributes except for pbs_version and sched_host

5.4.4. Reverting MoM to Defaults

• Remove prologue and epilogue (if any)
• Reset mom_priv/config to out-of-the-box defaults
• Optionally delete vnode definitions that may have been defined by test.
• Restart or send HUP signal to MoM:

• • If vnodes are deleted, MoM is restarted
  • Default is to HUP

• Vnodes are not deleted from the server object; it is the responsibility of the test writer to delete/create vnodes as needed at the beginning of the test

5.5 Generic Setup and Teardown Functions

SetUpClass Method

• Get test configuration parameters as a list of attributes

• Validate PTL parameters whether enabled or disabled

• Check whether or not the users exist

• Initialize servers, comms, schedulers, and MoMs 

SetUp Method

• Initialize process monitoring when requested
• Revert the servers, comms, schedulers, and MoMs to defaults, as described above in Overview of Reverting to Defaults

TearDown Method

• Stop process monitoring

TearDownClass Method

• Nothing is done inside TearDownClass()

6. Decorators

PTL provides decorators for use at the test suite level and at the test case level.  These decorators are defined in pbs_testsuite.py.

• timeout - Decorator to set timeout value of test case. (Default value : 600)
• requirements - Decorator to provide the cluster information required for a particular testcase
• skipOnCpuSet - Decorator to skip a test on a CpuSet system
• checkModule - Decorator to check if named module is available on the system and if not skip the test
• skip - Unconditionally skip a test.

How to Use Decorators

   @timeout(900)
   def test_smoke(Self):

7. Exception Handling

PTL defines a generic exception called "PtlException", which is inherited by numerous custom exceptions defined for multiple types of PBS operation failures.  PtlException defines generic errors raised by PTL operations. It sets a return value, a return code, and a message:

• return value is the output of the PTL operation
• return code is the error code returned by that operation
• message is the error message returned by the operation.

A post function and associated positional and named arguments are available to perform any necessary cleanup.

A bunch of exceptions are inherited from generic PtlException that are necessary for PBS operation error handling.

8. Related Links and References

Complete Doxygenated PTL Documentation

http://www.pbspro.org/ptldocs/

Repository

https://github.com/PBSPro/pbspro
https://github.com/PBSPro/pbspro/test

Nose Documentation

http://nose.readthedocs.io/en/latest/testing.html
http://pythontesting.net/framework/nose/nose-introduction/
http://nose.readthedocs.io/en/latest/plugins/writing.html

Python Documentation

https://docs.python.org/2.7/tutorial/

OSS Site Map

Developer Guide Pages

SaveSave

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

SaveSave

Save