1. Contents

1.1 Abbreviations

API: Application Programming Interface
CLI: Command-line Interface
fw: framework
PBS: PBS Professional
PTL: PbsTestLab

2. PTL Framework Components

PTL Framework Components

2.1 Interfaces to PBS

PTL executes tests with PBS operations using interfaces including the following:

PBS IFL-API
PBS commands
PBS configuration files
- sched_config
- mom_priv/config
- the holidays file
the PBS daemon log files

PTL provides two ways of running tests:

IFL API mode (PTL_API)
Command Line mode (PTL_CLI)

2.2 PBS Entities: Core Library

The PTL library provides PBS IFL operations through either SWIG wrappers or the PBS CLI. It contains classes that encapsulate the following PBS entities:

Server
Scheduler
MoM
Comm
Queue
Job
Reservation
Hook
Resource

The PTL libary includes utility classes to convert PBS data structures and attributes to Python lists, strings, and dictionaries. It provides APIs for high-level PBS operations to operate on PBS entities. The PTL library provides the below PBS entity objects used in writing tests:

Server Object: PBS server configuration and control
Scheduler Object: Container of scheduler-related properties
MoM Object: Container for MoM properties. Provides various MoM operations
Comm Object: PBS comm configuration and control
Job Object: PBS job attributes and resources
Reservation Object: PBS reservation attributes and resources
Queue Object: PBS Queue container. Holds queue attributes and pointer to server
Hook Object: PBS hook objects. Holds attribute information and pointer to server

2.3 Distributed Shell Utilities

The PBS shell utilities are a set of functions to run commands, copy files, get process information, and parse a PBS configuration on an arbitrary host. Below are a few examples of the functions that these utilities provide:

Get local or remote platform info
Get the path of the pbs.conf file
Set/unset environment or configuration variables in a pbs.conf file
Run a command on a host or list of hosts
Copy a file or directory to specified target hosts
Wrapper method to run PTL command
Generic functions include chmod, chown, chgrp, rm, mkdir, useradd, userdel, with remote host support, etc.

2.4 Log Utilities

Miscellaneous utilities to process and analyze PBS log files. Contains utilities to parse and report metrics from server, scheduler, MoM, and accounting logs. Metrics include number of jobs queued, run, ended, cycle duration, and in-scheduler duration. Below are a few examples of the utilities and functions:

Get the number of lines of a daemon log
Helper function to parse log file
Function to match a line in log file
Analyze server, scheduler, MoM, and accounting logs
Get PBS job information
Etc.

2.5 Testing Utilities

PTL provides testing utilities as modules that aid in building tests using the PTL objects of PBS entities.

These are in pbspro/test/fw/ptl/utils (https://github.com/PBSPro/pbspro/tree/master/test/fw/ptl/utils)

Below are a few of the testing utilities:

pbs_testsuite

This utility defines a main master test suite class called PBSTestSuite that defines generic setup, teardown, and logging functions to be used. It is taken as parent class for most tests. This class by default instantiates server, scheduler, MoM, and comm objects connected to localhost. It also provides a wrapper class to handle multiple service instances (i.e., MoM, server, and scheduler) and certain aiding functions such as a function to skip a test, a function to check modules available on the system, etc. This utility is explained in detail in a later section of this document.

pbs_cliutils

Generic system command interface utility that provides functions such as:

Check for the existence of a binary (a command) using the "which" command
Expand a relative path to an absolute path
Return a list of privileged ports in use on a given host
getting PTL logging levels
Etc.

pbs_covutils

Code coverage utilities. Below are a few of the functionalities provided by these utilities:

Set the coverage binary
Set the HTML generation utility
Set the coverage output directory
Set the coverage data directory
Initialize coverage
Capture the coverage parameters
Summarize the coverage output
Generate the HTML report
Etc.

pbs_procutils

Utilities to query process information. Below are a few of the utilities and functions provided:

Get process information from a process name or PID on a given host
Get PID's process state on host
Get process information reports for PID such as RSS and VSZ, along with the respective times
Background process monitoring tool
Etc

pbs_fileutils

Utility to walk a file using the result of 'head' or 'tail' on the local file system. Below are a few of the functions provided:

Return the file descriptor associated with the file being processed
Get the size of the file
Get the next n lines of the file from head or tail
Get the nth line of the file
Get a block of lines in a file
Etc.

2.6 Test Commands, Log Analyzer, Stat, and Filter

This layer contains the following test commands:

pbs_benchpress: Command used to run or list test suites and test cases
pbs_loganalyzer: Analyze PBS logs and return various throughput metrics
pbs_stat: Status and filter PBS objects based on select properties
pbs_swigify: Build IFL swig wrappers and copy them over to the library
pbs_as: Used by the library to impersonate a user for API job submission
pbs_config: Configure PBS: configure services, create and configure vnodes, etc.
pbs_py_spawn: Wrapper to the pbs_py_spawn API call
pbs_cov: Generates lcov/ltp (gcov) coverage analysis. Requires instrumented source files

The link below explains how to use the above test commands:
https://pbspro.atlassian.net/wiki/display/DG/Using+PTL+for+Testing#UsingPTLforTesting-Overviewofcommands

3. PTL Library: Objects and APIs

The PTL core library consists of objects encapsulating PBS entities. These entities are described in the following sections.

3.1 PBS Server Object

For PBS server configuration and control.
A server object is an instance of the Server class. A server object is a container for PBS server attributes, and uses wrappers for the IFL API to perform operations on the server. For example, you can use it to submit, status, delete, manage, etc. jobs, reservations, and configurations.
The server class also offers higher-level routines to ease testing. Examples: a function that reverts the PBS server configuration to prior known "out of the box" settings, a function to expect an attribute to match a given value as per an operation, and a function to return the count of anything queried in the PBS setup such as the number of nodes etc. based on certain conditions.
The class contains routines to act upon the PBS server daemon, such as starting, stopping, or restarting the PBS server daemon, and a function to match log messages in the daemon log files.
The server class also contains a dictionary that holds general configuration for the framework's operations, such as operation mode (PTL_CLI or PTL_API), maximum attempts to try while checking for a command in a test condition, and the interval between which this check of the command happens.
Below is a depiction of the Server class hierarchy:

Server Class

Server: The Server class
PBSService: Generic PBS service object to hold properties of PBS daemons
PBSObject: Generic PBS object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.2 PBS Scheduler Object

Container of Scheduler-related properties.
The Scheduler class contains routines to update scheduler configuration along with routines to act upon the PBS scheduler daemon, such as starting, stopping, or restarting the PBS scheduler daemon, and function to match log messages in the daemon log files.
This class also offers higher-level routines to ease testing. Examples: a function that reverts the PBS scheduler configuration to prior known "out of the box" settings, and a function to save daemon configurations.
Below is a depiction of the Scheduler class hierarchy:

Scheduler: The Scheduler Class
PBSService: Generic PBS service object to hold properties of PBS daemons
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.3 PBS MoM Object

Container for MoM properties.
The MoM class contains routines to update MoM attributes and act upon the PBS MoM daemon. Provides various MoM operations, such as creation, insertion, and deletion of vnodes.
This class also offers higher-level routines to ease testing. Examples: a function that reverts the PBS MoM configuration to prior known "out of the box" settings, and a function to save daemon configuration.
Below is a depiction of the MoM class hierarchy:

MoM Class

MoM: The MoM Class
PBSService: Generic PBS service object to hold properties of PBS daemons
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.4 PBS Comm Object

For PBS Comm configuration and control.
This class also offers higher-level routines to ease testing, for example: a function that reverts the PBS comm configuration to prior known "out of the box" settings, and a function to save daemon configurations.
Below is a depiction of the comm class hierarchy:

Comm Class

Comm: The Comm Class
PBSService: Generic PBS service object to hold properties of PBS daemons
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.5 PBS Job Object

Container for PBS Job attributes and resources.
This class defines certain default options of a PBS job including job name, job directory, mail_point, job priority, rerunnable, option to keep job output-error files on execution node, etc. These are overridden when any exclusive specifications are made after job object instantiation. Below are a few methods defined inside this class:

Set the sleep duration for this job
Set the executable and arguments to use for this job
Create a job script from a given body of text and put it in a temporary location
Customize the ``Variable_List`` job attribute to ``<user>``

Below is depiction of the job class hierarchy:

Job: The Job Class
ResourceResv: Generic PBS resource reservation (a job or advance or standing reservation)
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.6 PBS Reservation Object

Container for PBS Reservation attributes and resources. It initializes default start and end time; overridden when exclusive specifications are made.

Below is a depiction of the reservation class hierarchy:

Reservation Class

Reservation: The Reservation class
ResourceResv: Generic PBS resource reservation (a job or advance or standing reservation)
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.7 PBS Queue Object

Container for a PBS queue. Holds attributes of the queue and a pointer to the server. Also defines a method to revert queue attributes to defaults.
Below is a depiction of the queue class hierarchy:

Queue Class

Queue: The Queue Class
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

3.8 PBS Hook Object

Represents a PBS hook. Holds attribute information and a pointer to the server.
Below is a depiction of the class hierarchy:

Hook Class

Hook: The Hook class
PBSObject: Generic PBS Object encapsulating attributes and defaults
Object: The most basic type in built-in Python classes (_builtin_.py)

4. Communication of PBS Data in PTL

A test step involves sending the input data necessary for PBS operations to the PTL API and returning of the results of this PBS operation in test-understandable format. In this flow, based on the PTL operation mode, the below data formats are involved:

Python data structures: list, str, dict, tuple, list of dict, list of key=value pairs, etc.
PBS data structures: attrl, attropl, batch_status objects, etc. (defined in PBS)
PBS command argument mapping: mapped strings of PBS command-line arguments (defined in pbs_api_to_cli.py)

4.1 PBS Data Flow and Conversion in PTL

BatchUtils is a Utility class to create, convert, or display various PBS data structures. It is defined in PTL's core library (pbs_testlib) and is needed to convert the data as depicted below for PBS operations in PTL.

4.1.1 Data Flow and Conversion During Testing of PBS

Data Flow When Testing PBS

Testing PBS Operation in API Mode

The test provides the necessary PBS data in Python data structures to PTL API
This data is converted to PBS data structure format using functions defined in BatchUtils
The data is then passed to PBS IFL Wrapper functions to execute the test step

Testing PBS Operation in CLI Mode

The test provides the necessary PBS data in Python data structures to PTL API
This data is mapped to PBS command arguments using functions defined in BatchUtils
Then the whole PBS command is run by PTL

4.1.2 Data Flow and Conversion for Result of Testing PBS

Testing Result of PBS Operation in API Mode

The result of a PBS operationin API mode returns in PBS data structure format.
This result is converted to Python format using functions defined in BatchUtils.

Testing Result of PBS Operation in CLI Mode

The result of a PBS operation in CLI mode returns in the PBS format made up of a list of "key=value" pairs.
This result is converted to convenient Python data structure format using functions defined in BatchUtils.

4.2 Flow of Methods During Test Case Execution

Below is the flow of methods inside a PTL test suite that is inherited from the master test suite, named PBSTestSuite, when test cases are executed:

Method Flow in Test Suite

5. PBSTestSuite

5.1 Overview of PBSTestSuite

PBSTestSuite is a master test suite class that defines generic setup, teardown, and logging functions to be used by inherited test suites. This class instantiates server, scheduler, MoM, and comm objects connected to localhost. This class has functions that include deletion of all jobs and reservations, reversion of server and scheduler configuration to defaults, and checks to ensure that there is at least one CPU to schedule work on before running a test. This class also accepts a bunch of custom parameters based on the necessary test setup.

5.2 Parent Methods Defined in the Parent Class (PBSTestSuite)

Parent SetUpClass()
Parent SetUp()
Parent TearDown()
Parent TearDownClass()

5.3 Custom Methods Defined In the Test Suite

Custom SetUpClass()
Custom SetUp()
Custom TearDown()
Custom TearDownClass()

5.4 Reverting to Defaults

Server, scheduler, MoM, and queue each have a method to revert to defaults. Here is a description of what each one does:

5.4.1 Reverting Server to Defaults

Delete jobs and reservations
Unset all qmgr options that are different from out-of-the-box PBS options
Delete all non-default queues that the test may have created. Options exist to not delete the queues and instead to disable non-default queues
Delete all hooks that a test may have created. Options exist to instead disable hooks
Option exists to delete resourcedef file. Default is to not delete resourcedef file
Vnodes are not deleted from the server object; it is the responsibility of the test writer to delete/create vnodes as needed at the beginning of the test

5.4.2 Reverting Queue to Defaults

Unset queue attributes that are different from out-of-the-box queue attributes

5.4.3 Reverting Scheduler to Defaults

Clear any dedicated time that may have been set
Reset sched_config to out-of-the-box defaults
Send a HUP signal to the scheduler so that it re-reads its configuration file
Unset scheduler attributes except for pbs_version and sched_host

5.4.4. Reverting MoM to Defaults

Remove prologue and epilogue (if any)
Reset mom_priv/config to out-of-the-box defaults
Optionally delete vnode definitions that may have been defined by test.
Restart or send HUP signal to MoM:

- If vnodes are deleted, MoM is restarted
- Default is to HUP

Vnodes are not deleted from the server object; it is the responsibility of the test writer to delete/create vnodes as needed at the beginning of the test

5.5 Generic Setup and Teardown Functions

SetUpClass Method

Get test configuration parameters as a list of attributes
Validate PTL parameters whether enabled or disabled
Check whether or not the users exist
Initialize servers, comms, schedulers, and MoMs

SetUp Method

Initialize process monitoring when requested
Revert the servers, comms, schedulers, and MoMs to defaults, as described above in Overview of Reverting to Defaults

TearDown Method

Stop process monitoring

TearDownClass Method

Nothing is done inside TearDownClass()

6. Decorators

PTL provides decorators for use at the test suite level and at the test case level. These decorators are defined in pbs_testsuite.py.

timeout - Decorator to set timeout value of test case. (Default value : 600)
requirements - Decorator to provide the cluster information required for a particular testcase
skipOnCpuSet - Decorator to skip a test on a CpuSet system
checkModule - Decorator to check if named module is available on the system and if not skip the test
skip - Unconditionally skip a test.

How to Use Decorators

   @timeout(900)
   def test_smoke(Self):

7. Exception Handling

PTL defines a generic exception called "PtlException", which is inherited by numerous custom exceptions defined for multiple types of PBS operation failures. PtlException defines generic errors raised by PTL operations. It sets a return value, a return code, and a message:

return value is the output of the PTL operation
return code is the error code returned by that operation
message is the error message returned by the operation.

A post function and associated positional and named arguments are available to perform any necessary cleanup.

A bunch of exceptions are inherited from generic PtlException that are necessary for PBS operation error handling.

8. Related Links and References

SaveSave

Save