Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Boefje runonce functionality in scheduler #3906

Merged
merged 18 commits into from
Dec 6, 2024
Merged

Boefje runonce functionality in scheduler #3906

merged 18 commits into from
Dec 6, 2024

Conversation

jpbruinsslot
Copy link
Contributor

@jpbruinsslot jpbruinsslot commented Nov 28, 2024 •
edited
Loading

Changes

With the specification on the plugin definition of runon the BoefjeScheduler should be able to discern between task that needs to run once or that a schedule needs to be created, and based on what type of mutation (CREATE , UPDATE).

When a schedule or when a task needs to be run is laid out in this table #3879 (comment)

Note that the create_schedule attribute on a Scheduler always takes precedence. When this is set to False under no circumstances a schedule will be created. As is the case with the NormalizerScheduler.

Reviewer notes

Tests for the outcomes of the mentioned table should be made. This is as of yet untested, but can be reviewed, and qa'ed. UPDATE: tests have been added.

Issue link

Closes #3879

QA notes

Should be able to work with the following pr #3901

Reference this table #3879 (comment)

These paths should be tested, please additionally test some of the task to rescan and see what happens.

Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue.
  • I have written unit tests for the changes or fixes I made.
  • I have checked the documentation and made changes where necessary.
  • I have performed a self-review of my code and refactored it to the best of my abilities.
  • Tickets have been created for newly discovered issues.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

@jpbruinsslot jpbruinsslot added the mula Issues related to the scheduler label Nov 28, 2024
@jpbruinsslot jpbruinsslot self-assigned this Nov 28, 2024
@jpbruinsslot jpbruinsslot requested a review from a team as a code owner November 28, 2024 11:46
@jpbruinsslot jpbruinsslot linked an issue Nov 28, 2024 that may be closed by this pull request
Run boefje once at creation or update of ooi #3879
Closed
@jpbruinsslot jpbruinsslot marked this pull request as draft November 28, 2024 12:36
@jpbruinsslot jpbruinsslot changed the title Boefje runonce functionality in scheduler [WIP] Boefje runonce functionality in scheduler Nov 28, 2024
@jpbruinsslot jpbruinsslot mentioned this pull request Nov 28, 2024
Closed
@jpbruinsslot jpbruinsslot marked this pull request as ready for review December 2, 2024 11:01
@jpbruinsslot jpbruinsslot changed the title [WIP] Boefje runonce functionality in scheduler Boefje runonce functionality in scheduler Dec 2, 2024
@underdarknl
Copy link
Contributor

Looks like this should do the trick!

ammar92
ammar92 reviewed Dec 3, 2024
View reviewed changes
Copy link
Contributor

@ammar92 ammar92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work and tests 👍 Just a few things to consider, let me know what you think

mula/scheduler/models/plugin.py Outdated
@@ -19,3 +19,4 @@ class Plugin(BaseModel):
produces: list[str]
cron: str | None = None
interval: int | None = None
runon: list[str] | None = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runon doesn't exist in the dictionary

Suggested change
runon: list[str] | None = None
run_on: list[str] | None = None

I was also wondering, is it a str or should it be a list of predefined choices? But perhaps this could even be mapped to MutationOperationType?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could support both the string and the multiple option, however when we create a schema for the boefjes Im guessing it would be best to go for just the list.

mula/scheduler/schedulers/scheduler.py Outdated Show resolved Hide resolved
@noamblitz
Copy link
Contributor

noamblitz commented Dec 3, 2024 •
edited
Loading

This works well. How I tested:

  1. set SCHEDULER_PQ_GRACE_PERIOD=20
  2. run DNS boefje
  3. verify that it runs every 20 seconds
  4. add "runon": ["create","update"] to a boefje and remake
  5. verify that boefje only runs once (after updating scan level), and not every 20 seconds

@underdarknl underdarknl mentioned this pull request Dec 3, 2024
Open
@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Dec 6, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
17.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@underdarknl underdarknl merged commit 2320a41 into main Dec 6, 2024
21 of 22 checks passed
@underdarknl underdarknl deleted the feature/mula/create-schedule-overrides branch December 6, 2024 09:46
jpbruinsslot added a commit that referenced this pull request Dec 11, 2024
@jpbruinsslot
Merge branch 'main' into poc/mula/combined-schedulers
c40193f 
* main: (21 commits)
  Bump django from 5.0.9 to 5.0.10 in /rocky (#3940)
  Do not let enabling plugins affect the global plugin cache (#3944)
  Fix typing in more places and configure mypy to follow imports (#3932)
  Updates CWE archive to 4.16 (#3943)
  Report flaws (#3880)
  Translations update from Hosted Weblate (#3939)
  Fix report recipe API (#3942)
  Boefje runonce functionality in scheduler (#3906)
  fix: 🔨 do not store CDN findings (#3931)
  Dont check for Locations on local Ip's. (#3894)
  add unpkg.com to disallowed hostnames in CSP (#3927)
  Update website_discovery.py (#3921)
  Add export http boefje (#3901)
  Bump python-multipart from 0.0.9 to 0.0.18 in /bytes (#3925)
  Fix layout issues on scheduled reports page (#3930)
  Create scheduled report with zero objects selectable (#3907)
  Improve the KATalogus `/plugins` endpoint performance (#3892)
  Add bgp.jsonl and bgp-meta.json to .gitignore (#3928)
  Update pre-commit and all hooks (#3923)
  add support for detecting Lame dns delegations on ip ranges (#3899)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ammar92 ammar92 ammar92 left review comments

@underdarknl underdarknl underdarknl approved these changes

Assignees

@jpbruinsslot jpbruinsslot

Labels
mula Issues related to the scheduler
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Run boefje once at creation or update of ooi
4 participants
@jpbruinsslot @underdarknl @noamblitz @ammar92