Update file format and file name for data saved using save_configuration() method in PTL

Overview:

Currently cPickle is used to dump configuration using save_configuration() & load_configuration() is used to load it, which does not work as intended. Pickle is slow, unsafe and is not human readable. Storing configurations in json format would be a better option.

Forum Discussion: http://community.pbspro.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516

Technical Details:

Interface Update

In fw/ptl/lib/pbs_testlib.py,

• Interface: save_configuration()
  Synopsis: Update to save configuration in .json file. 
  Details: This method will encode configuration file contents in base64 format based on  & save it in a json file.

save_configuration() will continue to save the following from the PBS cluster:

Server configurations to be saved:

Commands output:
    qmgr print server
    qmgr print hook
    qmgr export hook <each hook> application/x-python default
    qmgr export hook <each hook> application/x-config default
    qmgr print resource
    qmgr list pbshook
    qmgr export pbshook <each pbshook> application/x-config default
    pbsnodes 

Schedular configurations to be saved:

Commands output:

    qmgr print sched

Configuration files:
    sched_priv/sched_config
    sched_priv/holidays
    sched_priv/resource_group
    sched_priv/dedicated_time
    multiple schedulers directories if any

Mom configurations to be saved:

Configuration files:
    mom_priv/config           (files of each hosts listed in pbsnodes)
    mom_priv/config.d/       (saves vnode definition files listed with "pbs_mom -s list")
    /etc/pbs.conf                 (all nodes)

In fw/ptl/lib/pbs_testlib.py

• Interface: load_configuration()
  Synopsis: Apply configuration saved by save_configuration()
  Details: This method will load json file that was saved by save_configuration & apply the corresponding changes.

Different aspects to be covered by this EDD are:

1. File name and path: A Single file to be used to save all the configurations, it includes all configuration of server, scheduler & mom. File name with the prefix as “pbsptl_conf” and suffix as “. json”. File path as temporary directory, using tempfile.mkstemp() to achieve this. Different versions of files to be maintained, every time save_configuration() is called.

E.g., pbsptl_conf_1.json – first iteration, pbsptl_conf_2.json – second iteration.

JSON file contents example:

2. File format: File contents can be encoded in base64 & that base64 is added to json file.

e.g, Suppose sched file contents are:

............

resources: "ncpus, mem, arch, host, vnode, aoe, eoe, cloud_scenario, cloud_node_image, cloud_node_instance_type, cloud_network"

............

It is encoded to base64 format = cmVzb3VyY2VzOiAibmNwdXMsIG1lbSwgYXJjaCwgaG9zdCwgdm5vZGUsIGFvZSwgZW9lLCBjbG91ZF9zY2VuYXJpbywgY2

                                                     xvdWRfbm9kZV9pbWFnZSwgY2xvdWRfbm9kZV9pbnN0YW5jZV90eXBlLCBjbG91ZF9uZXR3b3JrIg==

json file contents:

3. When user invokes pbs_benchpress, cleanup of tempdir is performed in case file with same name is present.

OSS Site Map

Project Documentation Main Page

Developer Guide Pages