Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standardize handling of storage and execution time quotas #1969

Merged
merged 13 commits into from
Jul 25, 2024
Merged

Standardize handling of storage and execution time quotas #1969

merged 13 commits into from
Jul 25, 2024

Commits on Jul 24, 2024

  1. Standardize storage and exec mins quota enforcement 

    - Check if both quotas are over before starting a crawl and during
the crawl, stop crawl gracefully if over
- Migrate existing stopped_quota_reached and skipped_quota_reached
crawl states to indicate which quota they relate to
- Use states that reflect both the action (stopped, skipped) and
which quota was over, checking storage quota first
    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    755a4cf View commit details
    Browse the repository at this point in the history

  2. Update crawl statuses and types in frontend

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    211fc3d View commit details
    Browse the repository at this point in the history

  3. Add back missing awaits

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    816734d View commit details
    Browse the repository at this point in the history

  4. Rename to ...time_quota_reached

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    0176537 View commit details
    Browse the repository at this point in the history

  5. Add nightly tests for storage quota

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4c3f28c View commit details
    Browse the repository at this point in the history

  6. Fixups

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    43fb351 View commit details
    Browse the repository at this point in the history

  7. Fix size constant label

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    aa0b401 View commit details
    Browse the repository at this point in the history

  8. Stop crawl gracefully if crawl size will go over storage quota

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    284d7e7 View commit details
    Browse the repository at this point in the history

  9. Remove fickle nightly test

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    854fbc0 View commit details
    Browse the repository at this point in the history

  10. Check all running crawl sizes, not just current crawl

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    8bf2192 View commit details
    Browse the repository at this point in the history

  11. Fetch org first

    @tw4l
    tw4l committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    a0d20ff View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Migrate lastCrawlState

    @tw4l
    tw4l committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    157e0c4 View commit details
    Browse the repository at this point in the history

  2. Improve docstring

    @tw4l
    tw4l committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d09d4b2 View commit details
    Browse the repository at this point in the history