Standardize handling of storage and execution time quotas #1969
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Check if both quotas are over before starting a crawl and during the crawl, stop crawl gracefully if over - Migrate existing stopped_quota_reached and skipped_quota_reached crawl states to indicate which quota they relate to - Use states that reflect both the action (stopped, skipped) and which quota was over, checking storage quota first
ikreymer
approved these changes
Jul 25, 2024
ikreymer
requested changes
Jul 25, 2024
ikreymer
added a commit
that referenced
this pull request
Jul 25, 2024
- instead of looking up storage and exec min quotas from oid, and loading an org each time, load org once and then check quotas on the org object - many times the org was already available, and was looked up again - storage and exec quota checks become sync - rename can_run_crawl() to more generic can_write_data(), optionally also checks exec minutes - follow up to #1969
ikreymer
approved these changes
Jul 25, 2024
ikreymer
added a commit
that referenced
this pull request
Jul 25, 2024
- instead of looking up storage and exec min quotas from oid, and loading an org each time, load org once and then check quotas on the org object - many times the org was already available, and was looked up again - storage and exec quota checks become sync - rename can_run_crawl() to more generic can_write_data(), optionally also checks exec minutes - follow up to #1969
ikreymer
added a commit
that referenced
this pull request
Jul 25, 2024
- instead of looking up storage and exec min quotas from oid, and loading an org each time, load org once and then check quotas on the org object - many times the org was already available, and was looked up again - storage and exec quota checks become sync - rename can_run_crawl() to more generic can_write_data(), optionally also checks exec minutes - typing: get_org_by_id() always returns org, or throws, adjust methods accordingly (don't check for none, catch exception) - typing: fix typo in BaseOperator, catch type errors in operator 'org_ops' - operator quota check: use up-to-date 'status.size' for current job, ignore current job in all jobs list to avoid double-counting - follow up to #1969
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1968
Changes:
stopped_quota_reachedandskipped_quota_reachedmigrated to new values that indicate which quota was reachedskipped_storage_quota_reachedorskipped_time_quota_reachedstopped_storage_quota_neededorstopped_time_quota_reachedstate as appropriateTo run the nightly tests, build the local backend and then run:
python -m pytest backend/test_nightly/test_storage_quota.pypython -m pytest backend/test_nightly/test_execution_minutes_quota.py