Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor schedulers #3793

Closed
wants to merge 15 commits into from
Closed

Conversation

jpbruinsslot
Copy link
Contributor

@jpbruinsslot jpbruinsslot commented Nov 4, 2024

Changes

  • Changes to folder structure to consolidate several folder and to create a clearer overview.

From:

├── alembic/
├── config/
├── connectors/
├── context/
├── models/
├── queues/
├── rankers/
├── schedulers/
├── server/
├── storage/
├── utils/
├── alembic.ini
├── __init__.py
├── __main__.py
├── app.py
└── version.py

To:

.
├── clients/
├── context/
├── models/
├── schedulers/
├── server/
├── storage/
├── utils/
├── __init__.py
├── __main__.py
├── app.py
└── version.py
  • Consolidate /queues and /schedulers endpoints, they are interchangeable. Meaning we only have /schedulers endpoint where we can reference and filter the available schedulers (mula/scheduler/server/handlers/schedulers.py)
  • Move schedulers configuration and settings to a postgres table. (mula/scheduler/models/scheduler.py). Meaning we now have schedulers persisted in a postgres table, before this was only available by referencing the in-memory schedulers dict.
  • Implement filtering of available schedulers from the rest API. Because we now have the schedulers present in a database we're able to use filter to get available schedulers (mula/scheduler/server/handlers/schedulers.py). For instance we can list all the schedulers for a specific organisation or we can query all the scheduler that have been created after a specific timestamp.
  • Change /pop endpoint to support popping of multiple tasks (batches), and add more filtering options (e.g. pop tasks for multiple organisations). We should now be able to create more sophisticated protocols to pop tasks from the queue.

Issue link

Closes: #3358

QA notes

Please add some information for QA on how to test the newly created code.


Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue.
  • I have written unit tests for the changes or fixes I made.
  • I have checked the documentation and made changes where necessary.
  • I have performed a self-review of my code and refactored it to the best of my abilities.
  • Tickets have been created for newly discovered issues.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.


Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

@jpbruinsslot jpbruinsslot added the mula Issues related to the scheduler label Nov 4, 2024
@jpbruinsslot jpbruinsslot self-assigned this Nov 4, 2024
@jpbruinsslot
Copy link
Contributor Author

Superseded by #3839

@jpbruinsslot jpbruinsslot deleted the feature/mula/refactor-schedulers branch December 10, 2024 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mula Issues related to the scheduler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor in-memory schedulers to postgresql table
1 participant