Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass the available worker/CPU info from the channel to the work producer code #2988

Open
harendra-kumar opened this issue Feb 18, 2025 · 0 comments
Labels
Milestone

Comments

@harendra-kumar
Copy link
Member

Consider the directory traversal code using parConcatIterate, the workers produce Left dirs and Right files. The workers need not produce Left dirs and can continue consuming those themselves unless there is a need for parallelism. Workers need to yield Left when there are idle workers (CPUs are not fully utilized). Somehow we need to make this feedback available to the workers.

One way is for the workers to periodically check the CPU utilization and yield more work for others to do if it is low and less if it is more. Another way may be to count active workers (not blocked for IO) and if active workers are less than a threshold then yield more work. But this may not work perfectly because even if there are workers idle CPUs might be busy doing some other work.

@harendra-kumar harendra-kumar added this to the 0.12.0 milestone Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant