Update file format and file name for data saved using save_configuration() method in PTL


Overview:

Currently cPickle is used to dump configuration using save_configuration() & load_configuration() is used to load it, which does not work as intended. Pickle is slow, unsafe and is not human readable. Storing configurations in json format would be a better option.

Forum Discussionhttp://community.pbspro.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516

Technical Details:

Interface Update

In fw/ptl/lib/pbs_testlib.py,

  • Interface: save_configuration()
    Synopsis: Update to save configuration in .json file. 
    Details: This method will encode configuration file contents in base64 format based on  & save it in a json file.

save_configuration() will continue to save the following from the PBS cluster:

Server configurations to be saved:

Commands output:
    qmgr print server
    qmgr print hook
    qmgr export hook <each hook> application/x-python default
    qmgr export hook <each hook> application/x-config default
    qmgr print resource
    qmgr list pbshook
    qmgr export pbshook <each pbshook> application/x-config default
    pbsnodes 


Schedular configurations to be saved:

Commands output:

    qmgr print sched


Configuration files:
    sched_priv/sched_config
    sched_priv/holidays
    sched_priv/resource_group
    sched_priv/dedicated_time
    multiple schedulers directories if any


Mom configurations to be saved:

Configuration files:
    mom_priv/config           (files of each hosts listed in pbsnodes)
    mom_priv/config.d/       (saves vnode definition files listed with "pbs_mom -s list")
    /etc/pbs.conf                 (all nodes)


In fw/ptl/lib/pbs_testlib.py

  • Interface: load_configuration()
    Synopsis: Apply configuration saved by save_configuration()
    Details: This method will load json file that was saved by save_configuration & apply the corresponding changes.

Different aspects to be covered by this EDD are:

1. File name and path: A Single file to be used to save all the configurations, it includes all configuration of server, scheduler & mom. File name with the prefix as “pbsptl_conf” and suffix as “. json”. File path as temporary directory, using tempfile.mkstemp() to achieve this. Different versions of files to be maintained, every time save_configuration() is called.

E.g., pbsptl_conf_1.json – first iteration, pbsptl_conf_2.json – second iteration.

JSON file contents example:

{

    "hostname1":

    {

        ..............................

        "qmgr_print_server" : "set server default_chunk.ncpus = 1
                                            set server flatuid = True
                                            set server resv_enable = True
                                            set server node_fail_requeue = 310"

         .............................

    }

}

2. All config and hook files content will be encoded in base64 format before adding to json file. Hooks file content will be added to same json file.

e.g, Suppose sched file contents are:

............

resources: "ncpus, mem, arch, host, vnode, aoe, eoe, cloud_scenario, cloud_node_image, cloud_node_instance_type, cloud_network"

............

It is encoded to base64 format = cmVzb3VyY2VzOiAibmNwdXMsIG1lbSwgYXJjaCwgaG9zdCwgdm5vZGUsIGFvZSwgZW9lLCBjbG91ZF9zY2VuYXJpbywgY2

                                                     xvdWRfbm9kZV9pbWFnZSwgY2xvdWRfbm9kZV9pbnN0YW5jZV90eXBlLCBjbG91ZF9uZXR3b3JrIg==

json file contents:



{

....

"hostname1":

    {

        ....

        "qmgr_print_server" : "set server default_chunk.ncpus = 1
                                            set server flatuid = True
                                            set server resv_enable = True
                                            set server node_fail_requeue = 310"

         ....

         "sched_config_file": "cmVzb3VyY2VzOiAibmNwdXMsIG1lbSwgYXJjaCwgaG9zdCwgdm5vZGUsIGFvZSwgZW9lLCBjbG91ZF9zY2VuYXJpbywgY2

                                           xvdWRfbm9kZV9pbWFnZSwgY2xvdWRfbm9kZV9pbnN0YW5jZV90eXBlLCBjbG91ZF9uZXR3b3JrIg=="

....

}

.

.

.

"PBS hooks":

    {

        "PBS_hookname.PY": "hook contents encoded in base64 format",

        "PBS_hookname.CF": "hook contents encoded in base64 format",

        .....

    }

"site hooks":

    {

        "site_hook_name.PY": "hook contents encoded in base64 format",

        "site_hook_name.CF": "hook contents encoded in base64 format",

        .....

     }


3. When user invokes pbs_benchpress, cleanup of tempdir is performed in case file with same name is present.











OSS Site Map

Project Documentation Main Page

Developer Guide Pages