Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: scheduler #275

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

Feature: scheduler #275

wants to merge 16 commits into from

Commits on Aug 26, 2024

  1. add scheduler

    superstar54 committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    b298b91 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Configuration menu
    Copy the full SHA
    f7ed561 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ed47f0b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a0dfea6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8d90643 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8669675 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    56ef506 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. Configuration menu
    Copy the full SHA
    7916e27 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Add checkpoint and docs

    superstar54 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    c36d72b View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. - Delete a workgraph data when it is finished

    - Pickup the Scheduler process instead of launching a new one.
    - submit a workgraph inside the scheduler
    - Move report from Scheduler process to the workgraph process
    superstar54 committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    9348801 View commit details
    Browse the repository at this point in the history
  2. handle failed task

    superstar54 committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    35aab9b View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Scheduler add workgraph subsriber

    The scheduler will listen to the task from scheduler_queue
    superstar54 committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    d267263 View commit details
    Browse the repository at this point in the history
  2. Support multiple scheduler.

    1) can run multiple runner (daemon) for the scheduler, each runner will listen to the `scheduler_queue`, and the prefetch_count is set to 1, thus each runner can only launch one Scheduler process.
    2) The scheduler process listen to the `workgraph_queue` to launch WorkGraph
    3) the scheduler recieve rpc call to launch WorkGrpah
    4) user can submit workgraph to the workgraph queue, or select the shceduler to run it by pk
    superstar54 committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    d35d63e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8299fe5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5a29c79 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Configuration menu
    Copy the full SHA
    eaa2f7e View commit details
    Browse the repository at this point in the history