Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/dns loc records #3349

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open

Feat/dns loc records #3349

wants to merge 40 commits into from

Conversation

noamblitz
Copy link
Contributor

@noamblitz noamblitz commented Aug 12, 2024

Changes

Adds LOC and GPOS records to octopoes and dns boefje.

Demo

Please add some proof in the form of screenshots or screen recordings to show (off) new functionality, if there are interesting new features for end-users.

QA notes

Please add some information for QA on how to test the newly created code.


Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue.
  • I have written unit tests for the changes or fixes I made.
  • I have checked the documentation and made changes where necessary.
  • I have performed a self-review of my code and refactored it to the best of my abilities.
  • Tickets have been created for newly discovered issues.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.


Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

@noamblitz noamblitz marked this pull request as ready for review August 12, 2024 16:18
@noamblitz noamblitz requested a review from a team as a code owner August 12, 2024 16:18
@noamblitz noamblitz marked this pull request as draft August 12, 2024 16:22
@noamblitz noamblitz marked this pull request as ready for review August 19, 2024 08:23
Copy link
Contributor

@ammar92 ammar92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! I just have a few remarks that could improve this even more. Let me know what you think

boefjes/boefjes/plugins/kat_dns/normalize.py Outdated Show resolved Hide resolved
boefjes/boefjes/plugins/kat_dns/normalize.py Outdated Show resolved Hide resolved
octopoes/octopoes/models/ooi/dns/records.py Outdated Show resolved Hide resolved
octopoes/octopoes/models/ooi/dns/records.py Outdated Show resolved Hide resolved
octopoes/octopoes/models/ooi/dns/records.py Outdated Show resolved Hide resolved
octopoes/octopoes/models/ooi/dns/records.py Outdated Show resolved Hide resolved
@stephanie0x00
Copy link
Contributor

Normalizer fails on the following error:

normalizer-1  | [2024-08-20 16:25:35 +0000] [16] [ERROR] [app] {"exc_info": true, "event": "An error occurred handling scheduler item[id=2a2f7780-3c15-4b0e-9b28-0dcd3c54fd36]", "level": "error", "timestamp": "2024-08-20T16:25:35.502693"}
normalizer-1  | Traceback (most recent call last):
normalizer-1  |   File "/app/boefjes/boefjes/app.py", line 242, in _start_working
normalizer-1  |     handler.handle(p_item.data)
normalizer-1  |   File "/app/boefjes/boefjes/job_handler.py", line 167, in handle
normalizer-1  |     results = self.job_runner.run(normalizer_meta, raw)
normalizer-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 86, in run
normalizer-1  |     return self._parse_results(normalizer_meta, results)
normalizer-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 94, in _parse_results
normalizer-1  |     for result in results:
normalizer-1  |   File "/app/boefjes/boefjes/plugins/kat_dns/normalize.py", line 54, in run
normalizer-1  |     responses.append(from_text("\n".join(lines[1:])))
normalizer-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normalizer-1  |   File "/usr/local/lib/python3.11/site-packages/dns/message.py", line 1678, in from_text
normalizer-1  |     return reader.read()
normalizer-1  |            ^^^^^^^^^^^^^
normalizer-1  |   File "/usr/local/lib/python3.11/site-packages/dns/message.py", line 1627, in read
normalizer-1  |     line_method(section_number)
normalizer-1  |   File "/usr/local/lib/python3.11/site-packages/dns/message.py", line 1462, in _header_line
normalizer-1  |     raise UnknownHeaderField
normalizer-1  | dns.message.UnknownHeaderField: The header field name was not recognized when converting from text
normalizer-1  |     into a message.

@stephanie0x00 stephanie0x00 added the 😸 Review/QA feedback Review/QA feedback provided label Aug 20, 2024
@ammar92
Copy link
Contributor

ammar92 commented Aug 21, 2024

Normalizer fails on the following error:

Could you perhaps share the input you're targetting (or raw file)? This would help with debugging

@noamblitz
Copy link
Contributor Author

I think we need to update dnspython, will check today.

@dekkers
Copy link
Contributor

dekkers commented Aug 22, 2024

Normalizer fails on the following error:

normalizer-1  | [2024-08-20 16:25:35 +0000] [16] [ERROR] [app] {"exc_info": true, "event": "An error occurred handling scheduler item[id=2a2f7780-3c15-4b0e-9b28-0dcd3c54fd36]", "level": "error", "timestamp": "2024-08-20T16:25:35.502693"}
normalizer-1  | Traceback (most recent call last):
normalizer-1  |   File "/app/boefjes/boefjes/app.py", line 242, in _start_working
normalizer-1  |     handler.handle(p_item.data)
normalizer-1  |   File "/app/boefjes/boefjes/job_handler.py", line 167, in handle
normalizer-1  |     results = self.job_runner.run(normalizer_meta, raw)
normalizer-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 86, in run
normalizer-1  |     return self._parse_results(normalizer_meta, results)
normalizer-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normalizer-1  |   File "/app/boefjes/boefjes/local.py", line 94, in _parse_results
normalizer-1  |     for result in results:
normalizer-1  |   File "/app/boefjes/boefjes/plugins/kat_dns/normalize.py", line 54, in run
normalizer-1  |     responses.append(from_text("\n".join(lines[1:])))
normalizer-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
normalizer-1  |   File "/usr/local/lib/python3.11/site-packages/dns/message.py", line 1678, in from_text
normalizer-1  |     return reader.read()
normalizer-1  |            ^^^^^^^^^^^^^
normalizer-1  |   File "/usr/local/lib/python3.11/site-packages/dns/message.py", line 1627, in read
normalizer-1  |     line_method(section_number)
normalizer-1  |   File "/usr/local/lib/python3.11/site-packages/dns/message.py", line 1462, in _header_line
normalizer-1  |     raise UnknownHeaderField
normalizer-1  | dns.message.UnknownHeaderField: The header field name was not recognized when converting from text
normalizer-1  |     into a message.

This seems to be #2805

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
😸 Review/QA feedback Review/QA feedback provided
Projects
Status: Blocked
Development

Successfully merging this pull request may close these issues.

5 participants