scheds: Introduce scx_tickless #1446
Open
+1,292
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
multics69
reviewed
Mar 2, 2025
Nope, I'll fix that, thanks for catching it! |
scx_tickless is a server-oriented scheduler, derived from scx_central,
designed to minimize OS noise and maximize performance isolation.
The scheduler works by routing all scheduling events through a pool of
primary CPUs assigned to handle these events. This allows disabling the
scheduler's tick on other CPUs, reducing OS noise.
Tasks are added to a global DSQ, and the primary CPUs distribute them
across the other "tickless" CPUs, sending preemption events via IPC in
case multiple tasks are competing for the same CPU.
In situations where the system is not overcommitted, tasks can run
uninterrupted on their assigned CPUs, leading to better performance
isolation and lower OS noise.
Typical use cases include cloud computing, virtualization and high
performance computing workloads. This scheduler is *not* designed for
latency-sensitive workloads.
NOTE: in order to effectively disable ticks on the "tickless" CPUs the
kernel must be booted with nohz_full.
Example:
- start a VM running a CPU-intensive workload:
$ vng -vr -- stress-ng -c 0
- on the host measure the rate of sched_tick events/sec:
$ bpftrace -e 'kprobe:sched_tick { @ticks++; } interval:s:1 { print(@ticks); clear(@ticks); }'
With the default scheduler:
@ticks: 1027
@ticks: 1021
@ticks: 1016
@ticks: 1016
@ticks: 1029
@ticks: 1024
@ticks: 1015
@ticks: 1022
@ticks: 1014
@ticks: 1056
...
With scx_tickless:
@ticks: 1000
@ticks: 1001
@ticks: 1001
@ticks: 1000
@ticks: 1001
@ticks: 1000
@ticks: 1000
@ticks: 1002
@ticks: 1000
@ticks: 1000
...
Signed-off-by: Andrea Righi <[email protected]>
htejun
approved these changes
Mar 3, 2025
| bool dispatched = false; | ||
|
|
||
| bpf_for_each(scx_dsq, p, SHARED_DSQ, 0) { | ||
| p = bpf_task_from_pid(p->pid); |
There was a problem hiding this comment.
Maybe add a comment explaining that this is working around a BPF issue and isn't needed on newer kernels?
| * non-primary CPUs were busy and a task was dispatched | ||
| * there. | ||
| */ | ||
| scx_bpf_kick_cpu(pick_primary_cpu(), SCX_KICK_PREEMPT); |
There was a problem hiding this comment.
Wouldn't it be cheaper for !primary cpu to dispatch directly if it's about to go idle anyway?
| !scx_bpf_dsq_nr_queued(SHARED_DSQ)) | ||
| continue; | ||
|
|
||
| scx_bpf_kick_cpu(cpu, SCX_KICK_PREEMPT); |
There was a problem hiding this comment.
I think it should be possible to call dispatch_all_cpus() directly from the timer.
sirlucjan
reviewed
Mar 3, 2025
|
@arighi Is there any chance that before the next stable release we will add this to scx_loader? Or does it not make sense in your opinion? |
|
@sirlucjan sure, I'll add the change to include it in scx_loader in this PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
scx_tickless is a server-oriented scheduler, derived from scx_central, designed to minimize OS noise and maximize performance isolation.
The scheduler works by routing all scheduling events through a pool of primary CPUs assigned to handle these events. This allows disabling the scheduler's tick on other CPUs, reducing OS noise.
Tasks are added to a global DSQ, and the primary CPUs distribute them across the other "tickless" CPUs, sending preemption events via IPC in case multiple tasks are competing for the same CPU.
In situations where the system is not overcommitted, tasks can run uninterrupted on their assigned CPUs, leading to better performance isolation and lower OS noise.
Typical use cases include cloud computing, virtualization and high performance computing workloads. This scheduler is not designed for latency-sensitive workloads.
NOTE: in order to effectively disable ticks on the "tickless" CPUs the kernel must be booted with nohz_full.
Example:
With the default scheduler:
With scx_tickless: