Overview:
Currently cPickle is used to dump configuration using save_configuration() & load_configuration() is used to load it, which does not work as intended. Pickle is slow, unsafe and is not human readable. Storing configurations in json format would be a better option.
Forum Discussion: http://community.pbspro.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516
Technical Details:
Interface Update
In fw/ptl/lib/pbs_testlib.py,
save_configuration() will continue to save the following from the PBS cluster:
Server configurations to be saved:
Commands output:
qmgr print server
qmgr print hook
qmgr export hook <each hook> application/x-python default
qmgr export hook <each hook> application/x-config default
qmgr print resource
qmgr list pbshook
qmgr export pbshook <each pbshook> application/x-config default
pbsnodes
Schedular configurations to be saved:
Commands output:
qmgr print sched
Configuration files:
sched_priv/sched_config
sched_priv/holidays
sched_priv/resource_group
sched_priv/dedicated_time
multiple schedulers directories if any
Mom configurations to be saved:
Configuration files:
mom_priv/config (files of each hosts listed in pbsnodes)
mom_priv/config.d/ (saves vnode definition files listed with "pbs_mom -s list")
/etc/pbs.conf (all nodes)
In fw/ptl/lib/pbs_testlib.py
Different aspects to be covered by this EDD are:
1. File name and path: A Single file to be used to save all the configurations, it includes all configuration of server, scheduler & mom. File name with the prefix as “pbsptl_conf” and suffix as “. json”. File path as temporary directory, using tempfile.mkstemp() to achieve this. Different versions of files to be maintained, every time save_configuration() is called.
E.g., pbsptl_conf_1.json – first iteration, pbsptl_conf_2.json – second iteration.
JSON file contents example:
{ "hostname1": { .............................. "qmgr_print_server" : "set server default_chunk.ncpus = 1 ............................. } } |
2. All config and hook files content will be encoded in base64 format before adding to json file. Hooks file content will be added to same json file.
e.g, Suppose sched file contents are:
............
resources: "ncpus, mem, arch, host, vnode, aoe, eoe, cloud_scenario, cloud_node_image, cloud_node_instance_type, cloud_network"
............
It is encoded to base64 format = cmVzb3VyY2VzOiAibmNwdXMsIG1lbSwgYXJjaCwgaG9zdCwgdm5vZGUsIGFvZSwgZW9lLCBjbG91ZF9zY2VuYXJpbywgY2
xvdWRfbm9kZV9pbWFnZSwgY2xvdWRfbm9kZV9pbnN0YW5jZV90eXBlLCBjbG91ZF9uZXR3b3JrIg==
json file contents:
{ .... "hostname1": { .... "qmgr_print_server" : "set server default_chunk.ncpus = 1 .... "sched_config_file": "cmVzb3VyY2VzOiAibmNwdXMsIG1lbSwgYXJjaCwgaG9zdCwgdm5vZGUsIGFvZSwgZW9lLCBjbG91ZF9zY2VuYXJpbywgY2 xvdWRfbm9kZV9pbWFnZSwgY2xvdWRfbm9kZV9pbnN0YW5jZV90eXBlLCBjbG91ZF9uZXR3b3JrIg==" .... } . . . "PBS hooks": { "PBS_hookname.PY": "hook contents encoded in base64 format", "PBS_hookname.CF": "hook contents encoded in base64 format", ..... } "site hooks": { "site_hook_name.PY": "hook contents encoded in base64 format", "site_hook_name.CF": "hook contents encoded in base64 format", ..... } |
3. When user invokes pbs_benchpress, cleanup of tempdir is performed in case file with same name is present.
Project Documentation Main Page
Ignore this. We may use it later for page characterization. |