Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Secondary Worker Node for PriorityQueue #9

Closed
DragonflyRobotics opened this issue Jun 20, 2022 · 1 comment
Closed

Add Secondary Worker Node for PriorityQueue #9

DragonflyRobotics opened this issue Jun 20, 2022 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@DragonflyRobotics
Copy link
Member

DragonflyRobotics commented Jun 20, 2022

Problem and Description
For the General Intelligence, the microphone transcription should always be running. But, if that is always in the queue, the other tasks like unsupervised detection and data download won't occur. The primary and daemon thread will be occupied. Is it possible to add a second worker node and a manager to handle task distribution and processing?

Solution
If we add a second __worker function to the PriorityQueue class as well as a manager to automatically distribute tasks once they are published, we can have more processes running with faster and more efficient execution. This also resolves the issue described above.

Alternatives
Multi-threading and Multi-processing managers were considered. In this idea, there would be a multiprocessing pool that would take tasks and assign them to a core manager. Each core manager takes the tasks and assigns them to a thread. This way, we get complete resource utilization. However, Python's GIL(Global Interpreter Lock) makes it arduous to implement. There is a separate issue for this: #10 .

@DragonflyRobotics
Copy link
Member Author

DragonflyRobotics commented Jun 23, 2022

A solution has been implemented for now that has as many threads as you desire as defined by config.json. This was on common eb1fa71!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant