Man Page for pbs-report
Prints PBS job statistics.
Synopsis
pbs-report [--age <seconds[:offset]>] [--account <account>] [--begin <yyyymmdd[:hhmm[ss]]> | -b <yyyymmdd[:hhmm[ss]]>]
[--count -c] [--cpumax <seconds>] [ --cpumin <seconds>] [--csv <character>] [--dept <department>] [--end <yyyymmdd[:hhmm[ss]]> | -e <yyyymmdd[:hhmm[ss]]>] [--exit <integer> | -x <integer>] [--explainwait] [--group <Linux group>] [--help] [--host <hostname>] [--inclusive] [--index <key> | -i <key>] [--man] [--negate <option>] [--package <solver>] [--point <yyyymmdd[:hhmm[ss]]>] [--queue <PBS queue>] [--range <span>] [--reslist] [--sched -t] [--sort <field>] [--time <option>] [--user <username>] [--verbose -v] [--vsort <field>] [--waitmax <seconds>] [--waitmin <seconds>] [--wall -w] [--wallmax <seconds>] [--wallmin <seconds>]
pbs-report --version
Description
Allows the PBS Administrator to generate a report of job statistics from the PBS accounting logfiles. Options to the pbs-report command control how jobs are selected for reporting and how reports are generated.
The pbs-report command is not available on Windows.
Permissions
This command can be run only by root.
Selecting Jobs For Reporting
Filtering Jobs by Dates or Times
--begin, --end, --range, --age, --point
--begin and --end
Work from hard date limits. Omitting either will cause the report to contain all data to either the beginning or the end of the accounting data. Unbounded date reports may take several minutes to run, depending on the volume of work logged.
Jobs are listed by start time only, regardless of whether end time is specified via --end or --inclusive.
--range <span>
A short-hand way of selecting a prior date range and will supersede --begin and --end.
--age <seconds[:offset]>
Allows the user to select an arbitrary period going back a specified number of seconds from the time the report is run. --age will silently supersede all other date options.
--point <yyyymmdd[:hhmm[ss]]>
Displays all jobs which were running at the specified point in time, and is incompatible with the other options. --point will produce an error if specified with any other date-related option.
Filtering Jobs by Attribute
--cpumax, --cpumin, --waitmax, --waitmin, --wallmax, --wallmin
A maximum value will cause any jobs with more than the specified amount to be ignored. A minimum value will cause any jobs with less than the specified amount to be ignored. All six options may be combined, though doing so will often restrict the filter such that no jobs can meet the requested criteria. Combine time filters for different time with caution.
Filtering Jobs by User or Department
--dept, --group, --user
--dept <department>
Allows for integration with an LDAP server and will generate reports based on department codes as queried from that server. If no LDAP server is available, department-based filtering and sorting will not function.
--group <Linux group>
Allows for filtering of jobs by primary group ownership of the submitting user, as defined by the operating system on which the PBS server runs.
--user <username>
Allows for explicit naming of users to be included.
It is possible to specify a list of values for these filters, by providing a single colon-concatenated argument or using the option multiple times, each with a single value.
Filtering Jobs by Job Property
--host, --exit, --package, --queue
--host <hostname>
Allows for filtering of jobs based on the host on which the job was executed.
--exit <integer> or -x <integer>
Allows for filtering of jobs based on the job exit code.
--package <solver>
Allows for filtering of jobs based on the software package used in the job. This option will only function when a package-specific custom resource is defined for the PBS server and requested by the jobs as they are submitted.
--queue <PBS queue>
Allows for filtering of jobs based on the queue in which the job finally executed. With the exception of --exit, it
is possible to specify a list of values for these filters, by providing a single colon-concatenated argument or
using the option multiple times, each with a single value.
Filtering Jobs by Account String
--account <account>
This option allows the user to filter jobs based on an arbitrary, user-specified job account string. The content
and format of these strings is site-defined and unrestricted; it may be used by a custom job front-end which
enforces permissible account strings, which are passed to qsub with qsub’s -A option.
Negating Filters
--negate <filter>
The --negate option allows for logical negation of one or more specified filters. Only the account, dept, exit, group, host, package, queue, and user filters may be negated. If a user is specified with --user, and the ‘--negate user’ option is used, only jobs not belonging to that user are included in the report. Multiple report filters may be negated by providing a single colon-concatenated argument or using --negate multiple times, each with a single value.
Generating Reports
Several report types can be generated, each indexed and sorted according to the user’s needs.
--verbose
Generates a wide tabular output with detail for every job selected. It can be used to generate output for import to a spreadsheet. Verbose reports may be sorted on any field using the --vsort option. Default: summary report only.
--reslist
Generates a tabular output with detail on resources requested for every job selected. Resource list reports may be sorted on any field using the --vsort option. Default: summary report only.
--inclusive
Allows a user to require that the job’s start time also falls within the date range. Jobs are listed by start time only, regardless of whether end time is specified via --end or --inclusive.
--index <key>
Allows specification of the field on which data in the summary should be grouped. Fields listed in the option description are mutually exclusive. The selected field is the left-most column of the summary report output. One value may be selected as an index while another is selected for sorting. However, since index values are mutually exclusive, the only sort options which may be used (other than the index itself) are account, cpu, jobs, suspend, wait, and wall. If no sort order is selected, the index is used as the sort key for the summary.
--sort <field>
Allows the user to specify a field on which to sort the summary report. It operates independently of the sort field for verbose reports (see --vsort ). See the description for --index for how the two options interact.
--vsort <field>
Allows the user to specify a field on which to sort the verbose report. It operates independently of the sort field for summary reports (see --sort ).
Options to pbs-report
--age <seconds[:offset]> or -a <seconds[:offset]>
Report age in seconds. If an offset is specified, the age range is taken from that offset backward in time, otherwise a zero offset is assumed. The time span is from (now - age - offset) to (now - offset). This option silently supersedes --begin, --end, and --range.
--account <account>
Limit results to those jobs with the specified account string. Multiple values may be concatenated with colons or specified with multiple instances of --account.
--begin <yyyymmdd[:hhmm[ss]]> or -b <yyyymmdd[:hhmm[ss]]>
Report begin date and optional time. Default: most recent log data. --begin and --end work from hard date limits. Omitting either will cause the report to contain all data to either the beginning or the end of the accounting data. Unbounded date reports may take several minutes to run, depending on the volume of work logged.
Jobs are listed by start time only, regardless of whether end time is specified via --end or --inclusive.
--count -c
Display a numeric count of matching jobs. Currently only valid with --cpumax for use in monitoring rapidlyexiting jobs.
--cpumax <seconds>
Filter out any jobs which have more than the specified number of CPU seconds.
--cpumin <seconds>
Filter out any jobs which have less than the specified number of CPU seconds.
--dept <department> or -d <department>
Limit results to those jobs whose owners are in the indicated department . Default: any. This option only works in conjunction with an LDAP server which supplies department codes. See also the --group option. Multiple values may be concatenated with colons or specified with multiple instances of --dept.
--end <yyyymmdd[:hhmm[ss]]> or -e <yyyymmdd[:hhmm[ss]]>
Report end date and optional time. Default: most recent log data. --begin and --end work from hard date limits. Omitting either will cause the report to contain all data to either the beginning or the end of the accounting data. Unbounded date reports may take several minutes to run, depending on the volume of work logged.
Jobs are listed by start time only, regardless of whether end time is specified via --end or --inclusive.
--exit <integer> | -x <integer>
Limit results to jobs with the specified exit status. Default: any.
--explainwait
Print a reason for why jobs had to wait before running.
--group <group> or -g <group>
Limit results to the specified group name. Group is defined by the operating system. Multiple values may be concatenated with colons or specified with multiple instances of --group.
--help -h
Prints a brief help message and exits.
--host <execution host> or -m <execution host>
Limit results to the specified execution host. Multiple values may be concatenated with colons or specified with multiple instances of --host.
--inclusive <key>
Limit results to jobs which had start times in the range.
Jobs are listed by start time only, regardless of whether end time is specified via --end or --inclusive.
--index <key> or -i <key>
Field on which to index the summary report. Default: user. Valid values include: date, dept, host, package, queue, user.
--man
Prints the manual page and exits.
--negate <option> or -n <option>
Logically negate the selected options; print all records except those that match the values for the selected criteria.
Default: unset. Valid values: account, dept, exit, group, host, package, queue, user. Defaults cannot be negated, only options explicitly specified are negated. Multiple values may be concatenated with colons or specified with multiple instances of --negate.
--package <package> or -p <package>
Limit results to the specified software package. Multiple values may be concatenated with colons or specified with multiple instances of --package. Valid values are can be seen by running a report with the --index package option. This option keys on custom resources requested at job submission time. Sites not using such custom resources will have all jobs reported under the catchall None package with this option.
--point <yyyymmdd[:hhmm[ss]]>
Print a report of all jobs which were actively running at the point in time specified. This option cannot be used with any other date or age option.
--queue <queue name> or -q <queue name>
Limit results to the specified queue. Multiple values may be concatenated with colons or specified with multiple instances of --queue. Note that if specific queues are defined via the @QUEUES line in PBS.pm, then only those queues are displayed. Leaving that parameter blank allows all queues to be displayed.
--range <period> or -r <period>
Time period used is period before now. For example, if the period given is “week”, pbs-report looks at all jobs which have finished and which were running any time from a week ago to now. Default: all. Valid values for period are today, week, month, quarter, and year. This option silently supersedes --begin and --end, and is superseded by --age.
--reslist
Include resource requests for all matching jobs. This option is mutually exclusive with --verbose.
--sched -t
Generate a brief statistical analysis of Scheduler cycle times. No other data on jobs is reported.
--sort <field> or -s <field>
Field by which to sort reports. Default: user. Valid values are cpu, date, dept, host, jobs, package, queue, suspend (aka muda), wait, and wall.
--time <option>
Valid values: “full”, “partial”. Used to indicate how time should be accounted. The default of “full” means that entire job’s CPU and wall time is counted in the report if the job ended during the report’s date range. With the “partial” option, only CPU and wall time during the report’s date range are counted.
By default, time is credited at the point when the job ended. This can be changed using the --inclusive option. For a job which ended a few seconds after the report range begins, this can cause significant overlap, which may boost results. During a sufficiently large time frame, this overlap effect is negligible and may be ignored. This value for --time should be used when generating monthly usage reports. With “partial”, any CPU or wall time accumulated prior to the beginning of the report is ignored. “partial” is intended to allow for more accurate calculation of overall cluster efficiency during short time spans during which a significant overlap effect can skew results. See --inclusive.
--user <username> or -u <username>
Limit results to the specified username. Multiple values may be concatenated with colons or specified with multiple instances of --user.
--verbose -v
Include attributes for reported jobs. Subjobs are shown, but not job arrays. Default: no attributes.
--version
The pbs-report command returns its PBS version information and exits. This option can only be used alone.
--vsort <field>
Field by which to sort the verbose output section reports. Default: jobid. Valid values are cpu, date, exit, host, jobid, jobname, mem, name, package, queue, scratch, suspend, user, vmem, wall, wait. If neither --verbose nor --reslist is specified, --vsort is silently ignored. The scratch sort option is available only for resource reports ( --reslist ).
--waitmax <seconds>
Filter out any jobs which have more than the specified wait time in seconds.
--waitmin <seconds>
Filter out any jobs which have less than the specified wait time in seconds.
--wallmax <seconds>
Filter out any jobs which have more than the specified wall time in seconds.
--wallmin <seconds>
Filter out any jobs which have less than the specified wall time in seconds.
--wall -w
Use the walltime resource attribute rather than wall time calculated by subtracting the job start time from end time. The walltime resource attribute does not accumulate when a job is suspended for any reason, and thus may not accurately reflect the local interpretation of wall time.
Examples
“How much in the way of resources did every job this month waiting more than 10 minutes request?”
pbs-report --range month --waitmin 600 --reslist
This information might be valuable to determine if some simple resource additions (e.g. more memory or more disk) might increase overall throughput of the cluster.
Statistical Analysis
At the bottom of the summary statistics, prior to the job set summary, is a statistical breakdown of the values in each column. Example:
# of Total Total Average
Date jobs CPU Time Wall Time Efcy. Wait Time
--------- ------ --------- --------- ----- ---------
TOTAL 1900 10482613 17636290 0.594 1270
Minimum 4 4715 13276 0.054 221
Maximum 162 1399894 2370006 1.782 49284
Mean 76 419304 705451 0.645 2943
Deviation 41 369271 616196 0.408 9606
Median 80 242685 436724 0.556 465
This summary should be read in column format. The values each represent a statistical data point in the column. For instance, while the minimum number of jobs run in one day was 4 and the maximum 162, these values do not correlate to the 4715 and 1399894 CPU seconds listed as minima and maxima.
In the Job Set Summary section, the values should be read in rows, as shown here:
Standard
Minimum Maximum Mean Deviation Median
------- ------- ---- --------- -------
CPU time 0 18730 343 812 0
Wall time 0 208190 8496 19711 93
Wait time 0 266822 4129 9018 3
These values represent aggregate statistical analysis for the entire set of jobs included in the report. The values in the prior summary represent values over the set of totals based on the summary index (e.g. Maximum and Minimum are the maximum and minimum totals for a given day/user/department, rather than an individual job. The job set summary represents an analysis of all individual jobs.
Cluster Monitoring
The --count and --cpumax functions are intended to allow an administrator to periodically run this script to monitor for jobs which are exiting rapidly, representing a potential global error condition causing all jobs to fail. It is most useful in conjunction with --age, which allows a report to span an arbitrary number of seconds backward in time from the current moment. A typical set of options would be “--count --cpumax 30 --age 21600”, which would show a total number of jobs which consumed less than 30 seconds of CPU time within the last six hours.
Configuration for the pbs-report Command
We recommend that before you first use pbs-report, you tune the configuration to match the local site by editing the file PBS_EXEC/lib/pm/PBS.pm. For example, if you want to see solver usage, add your solvers to the “our @PACKAGES” field on line 56 in this file.
Standard Error
The pbs-report command writes a diagnostic message to standard error for each error occurrence.
Exit Status
Zero
Upon successful processing of all operands.
Greater than zero
If the pbs-report command fails to process any operand.