Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

catch the schema mismatch error and produce an error raw file #3995

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

catch the schema mismatch error and produce an error raw file #3995

wants to merge 5 commits into from

Conversation

underdarknl
Copy link
Contributor

Changes

Catch the schema validation error and produce a raw file with the error mimetype.

Issue link

Closes #3993

Demo

Please add some proof in the form of screenshots or screen recordings to show (off) new functionality, if there are interesting new features for end-users.

QA notes

Please add some information for QA on how to test the newly created code.

Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue.
  • I have written unit tests for the changes or fixes I made.
  • I have checked the documentation and made changes where necessary.
  • I have performed a self-review of my code and refactored it to the best of my abilities.
  • Tickets have been created for newly discovered issues.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

@underdarknl underdarknl requested a review from a team as a code owner December 31, 2024 09:18
ammar92
ammar92 previously approved these changes Jan 3, 2025
View reviewed changes
@stephanie0x00
Copy link
Contributor

During QA when no Ports are provided the same error is still shown, and it's still not possible to get the raw file. When you apply a port and reschedule the task, the task does complete and the raw file is accessible.

boefje-1  | HTTP Request: GET http://octopoes_api/aa/object?reference=IPAddressV4%7Cinternet%7C134.209.85.72&valid_time=2025-01-06%2009%3A01%3A49.113387%2B00%3A00 "HTTP/1.1 200 OK"
boefje-1  | HTTP Request: GET http://katalogus:8000/v1/organisations/aa/nmap-ports/settings "HTTP/1.1 200 OK"
boefje-1  | {"exc_info": true, "event": "Error running boefje due to settings/schema mismatch nmap-ports[8aaac702-b0b0-4d09-90af-4430dbb4fcd3]", "level": "error", "timestamp": "2025-01-06T09:01:49.163293"}
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 86, in get_environment_settings
boefje-1  |     validate(instance=new_env, schema=schema)
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 1332, in validate
boefje-1  |     raise error
boefje-1  | jsonschema.exceptions.ValidationError: 'PORTS' is a required property
boefje-1  | 
boefje-1  | Failed validating 'required' in schema:
boefje-1  |     {'title': 'Arguments',
boefje-1  |      'type': 'object',
boefje-1  |      'properties': {'PORTS': {'title': 'PORTS',
boefje-1  |                               'maxLength': 2048,
boefje-1  |                               'type': 'string',
boefje-1  |                               'pattern': '^((6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})|(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})-(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4}))$|^((6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})|(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})-(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4}))(,((6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})|(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})-(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})))+$',
boefje-1  |                               'description': 'Specify the ports that need '
boefje-1  |                                              'to be scanned (nmap format). '
boefje-1  |                                              'Single ports are comma '
boefje-1  |                                              'separated, port ranges can '
boefje-1  |                                              'be specified using the dash '
boefje-1  |                                              'symbol. For example: '
boefje-1  |                                              '22,111,137,80-100 will scan '
boefje-1  |                                              'ports 22, 111, 137 and the '
boefje-1  |                                              'port range 80 up to 100.'}},
boefje-1  |      'required': ['PORTS']}
boefje-1  | 
boefje-1  | On instance:
boefje-1  |     {}
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 139, in handle
boefje-1  |     boefje_meta.environment = get_environment_settings(boefje_meta, plugin.boefje_schema)
boefje-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 88, in get_environment_settings
boefje-1  |     raise SettingsNotConformingToSchema(boefje_meta.boefje.id, e.message) from e
boefje-1  | boefjes.storage.interfaces.SettingsNotConformingToSchema: ('nmap-ports', "'PORTS' is a required property")
boefje-1  | {"exc_info": true, "event": "An error occurred handling scheduler item[id=8aaac702-b0b0-4d09-90af-4430dbb4fcd3]", "level": "error", "timestamp": "2025-01-06T09:01:49.165450"}
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 86, in get_environment_settings
boefje-1  |     validate(instance=new_env, schema=schema)
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/jsonschema/validators.py", line 1332, in validate
boefje-1  |     raise error
boefje-1  | jsonschema.exceptions.ValidationError: 'PORTS' is a required property
boefje-1  | 
boefje-1  | Failed validating 'required' in schema:
boefje-1  |     {'title': 'Arguments',
boefje-1  |      'type': 'object',
boefje-1  |      'properties': {'PORTS': {'title': 'PORTS',
boefje-1  |                               'maxLength': 2048,
boefje-1  |                               'type': 'string',
boefje-1  |                               'pattern': '^((6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})|(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})-(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4}))$|^((6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})|(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})-(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4}))(,((6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})|(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})-(6553[0-5]|655[0-2]\\d|65[0-4]\\d{2}|6[0-4]\\d{3}|[1-5]\\d{4}|\\d{1,4})))+$',
boefje-1  |                               'description': 'Specify the ports that need '
boefje-1  |                                              'to be scanned (nmap format). '
boefje-1  |                                              'Single ports are comma '
boefje-1  |                                              'separated, port ranges can '
boefje-1  |                                              'be specified using the dash '
boefje-1  |                                              'symbol. For example: '
boefje-1  |                                              '22,111,137,80-100 will scan '
boefje-1  |                                              'ports 22, 111, 137 and the '
boefje-1  |                                              'port range 80 up to 100.'}},
boefje-1  |      'required': ['PORTS']}
boefje-1  | 
boefje-1  | On instance:
boefje-1  |     {}
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/app.py", line 251, in _start_working
boefje-1  |     handler.handle(p_item.data)
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 139, in handle
boefje-1  |     boefje_meta.environment = get_environment_settings(boefje_meta, plugin.boefje_schema)
boefje-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 88, in get_environment_settings
boefje-1  |     raise SettingsNotConformingToSchema(boefje_meta.boefje.id, e.message) from e
boefje-1  | boefjes.storage.interfaces.SettingsNotConformingToSchema: ('nmap-ports', "'PORTS' is a required property")
boefje-1  | HTTP Request: GET http://scheduler:8000/tasks/8aaac702-b0b0-4d09-90af-4430dbb4fcd3 "HTTP/1.1 200 OK"
boefje-1  | HTTP Request: PATCH http://scheduler:8000/tasks/8aaac702-b0b0-4d09-90af-4430dbb4fcd3 "HTTP/1.1 200 OK"

@ammar92
Copy link
Contributor

ammar92 commented Jan 7, 2025

During QA when no Ports are provided the same error is still shown, and it's still not possible to get the raw file. When you apply a port and reschedule the task, the task does complete and the raw file is accessible.

The error is shown as expected (as a direct output of logger.exception), but the missing raw file isn't. After some investigation, I can see why it fails. The problem is that saving the raw file happens a bit later in another try-catch-finally-clause. But this implementation raises this exception too early which prematurely ends the execution flow. I'll take a look on how to handle this properly and refactor it

@ammar92 ammar92 self-assigned this Jan 7, 2025
@ammar92
Copy link
Contributor

ammar92 commented Jan 8, 2025

I just fixed the implementation. After QA I can confirm this works now, the task gets the appropriate 'failed' state with the used schema for settings as the raw file

@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Jan 8, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
62.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ammar92 ammar92 ammar92 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@ammar92 ammar92

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

jsonschema.exceptions.ValidationError: 'PORTS' is a required property
3 participants
@underdarknl @stephanie0x00 @ammar92