Motivation: Scheduler sends job attribute updates for certain attributes to the server. These updates can become a performance bottleneck, affecting scheduling speed. The most problematic ones are the updates related to “job can’t run”, where scheduler updates the job comment, possibly accrue_type, for each job that’s not run, even if it might run those jobs in the very next cycle. So, throttling such updates sounds reasonable, and can speed up the scheduler greatly. By reducing the total number of requests sent to the server, it should also help the server be more responsive to other more important requests.
External changes:
New sched attribute “attr_update_freq”: This will determine the number of cycles after which scheduler will send “job can’t run” related attribute updates to the server. These attributes are as follows:
comment
estimated.exec_vnode
estimated.start_time
Default: The default value will be 1, i.e - scheduler will send updates every cycle, just like today. For sites which see a large volume of jobs, admins might want to increase this. With a workload of 100k jobs every cycle, 50k of which wouldn’t run, I saw a 3x+ performance boost when setting this value to 5.
Exception: If “accrue_type” needs to be updated for a job, then scheduler will ignore the throttling window & send all attribute updates for that job immediately so that eligible time is accrued accurately.
To ameliorate this, I’m also proposing that we change the default accrue_type of jobs to ‘eligible_time’ instead of ‘initial_time’. It seems like we already document “eligible_time” as the default for accrue_type, so this shouldn’t need a separate design change document.
Permissions: Manager write only, everyone can read
Implications:
Depending on the configured frequency of updates, there will be delays associated with when attributes like job comment get updated, which tells users why their job isn’t running. Since the sched cycles will be faster, this delay might not be much, but admins should keep this in mind and figure out what the value of this attribute works best for their site.
Can be configured per scheduler in a multi-sched scenario
All “job run” related attribute updates, like pset, walltime for STF jobs, etc., will be sent like before.
We are removing pset soon (Deprecate 'pset' job attribute), so hopefully “job run” updates won’t be too prolific.
Technical details:
Scheduler will simply send “job can’t run” type updates every N cycles without caching them internally as these jobs will be looked at again the next cycle.