Forum discussion link :http://community.pbspro.org/t/external-design-document-for-pp-824-cray-ramp-rate-limiting/693
...
Parameter Name | Default value | Description |
|---|---|---|
| power_ramp_rate_enable | False | Enabling would make PBS perform ramp rate limiting across the PBS cluster running on a CRAY CLE 6.0 platform. Nodes will be ramped-up and kept at sleep state C1 and for ramp down nodes will be put to sleep state C6. |
| power_on_off_enable | False | Enabling would make PBS power on and off nodes on the nodes where node attribute poweroff_eligible is true. |
| node_idle_limit | 1800 | How long any node should be left idle before it to be considered for powering down or ramp down. |
| min_node_down_delay | 1800 | The time limit before a powered-off node can be considered to be brought up. |
| max_jobs_analyze_limit | 100 | The limit indicating maximum number of jobs that are analyzed for power on/ramp-up. The jobs considered here are those which have estimated start_time and exec_vnode updated on them. To have these attributes updated one should have strict_ordering set to true and submit jobs with walltime. |
| max_concurrent_nodes | 105 | Defines how many nodes can be power on/off or ramped up/down at a time. For ramp rate, while stepping up or down sleep states, hook will sleep X seconds (where 1<=X<=10) between each level of sleep state. If a node supports 5 levels of sleep states, in worst case scenario hook can wait for 50 seconds for single node. So while increasing the value of this attribute one should also consider increasing the PBS_Power hook frequency and alarm time so that hook instances do not overlap or timeout. |
- Interface 3: New node attribute: poweroff_eligible
- Change control: Stable
- Synopsis: Node attribute for power control.
- Details: This new node attribute will control if a node can be allowed to power off or not.
- PBS type: Boolean
- Python type: Boolean
- Default value: False
- Manager has set permission. All have read permission.
- To modify the default value use qmgr:
- qmgr -c "set node <node_name> poweroff_eligible=True"
- Interface 4: New node attribute: last_state_change_time
- Change control: Stable
- Synopsis: Read only node attribute to capture timestamp.
- Details: This new node attribute will be updated with time stamp when the node changes from its current state to a new state.
- Managers and Operators have read permission.
- Node status command pbsnodes will convert internal date format (seconds since epoch) to human readable format and display the value of this attribute in “MON DD YY HH:MM:SS” format.
- PBS type: long
- Python type: int
- Interface 5: New node attribute: last_used_time
- Change Control: Stable
- Synopsis: Read only node attribute to capture timestamp.
- Details: This new node attribute will be updated with time stamp at the end of any job or reservation.
- If node is released early from a running job this timestamp gets updated.
- Node status command pbsnodes will convert internal date format (seconds since epoch) to human readable format and display the value of this attribute in "MON DD YY HH:MM:SS" format.
- Attribute will be reset when node is ramped up.
- Managers and Operators have read permission.
- For vnodes this attribute will be updated for the first time with the current timestamp when they are created or when the nodes are rebooted.
- This attribute can now be used in sched_config as a node_sort_key. This will help sort the nodes based on their last used time.
- PBS type: long
- Python type: int
- Example:
- node_sort_key: "last_used_time HIGH"
- node_sort_key: "last_used_time LOW"
- Interface 6: New node state: sleep
- Change Control: Stable
- Synopsis: New node state that shows node is put down by PBS.
- Details: This new node state will be set when nodes are ramped down or powered-off by PBS via power ramp rate limiting or power on/off feature.
- A server periodic hook (pbs hook PBS_power provided as part of PBS package) runs every $freq seconds and takes list of vnodes to power ramp down/power-off the nodes and marks them in new sleep node state.
- At most max_concurrent_power_limit nodes will be ramped down/powered-off every freq seconds, freq being the server periodic hook frequency.
- Scheduler can consider the nodes in sleep state to run jobs now.
- Server periodic hook can ramp-up/power-on the nodes which are in sleep state based on the requirement. Requirement is calculated based on analyzing the jobs estimated start time and the exec_vnode list.
...