Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Towards better inference: bits → nibbles #3808

Open
wants to merge 189 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 177 commits
Commits
Show all changes
189 commits
Select commit Hold shift + click to select a range
ae00a8e
Introducing nibbles
originalsouth Aug 27, 2024
c90fcb0
Prototyping
originalsouth Aug 28, 2024
d57cf19
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Sep 18, 2024
64ece62
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Sep 18, 2024
bba22a3
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Sep 18, 2024
0896eba
set default in model
noamblitz Sep 19, 2024
964b89b
remove default bit
noamblitz Sep 19, 2024
5915f03
fix test
noamblitz Sep 19, 2024
ed7be58
Fix Octopoes tests for patch related changes
originalsouth Sep 19, 2024
efa3c97
Merge branch 'set-default-risk-in-model' of github.com:minvws/nl-kat-…
originalsouth Sep 19, 2024
663a9bb
Fix Octopoes tests for patch related changes II
originalsouth Sep 19, 2024
bd78ed9
Merge branch 'main' into set-default-risk-in-model
originalsouth Sep 19, 2024
b5ba90a
Fix Octopoes tests for patch related changes III
originalsouth Sep 19, 2024
f885652
Merge branch 'set-default-risk-in-model' of github.com:minvws/nl-kat-…
originalsouth Sep 19, 2024
b05283e
Prevent race conditions between Octopoes' event manager and the sched…
originalsouth Sep 19, 2024
06d1080
Merge branch 'main' into set-default-risk-in-model
underdarknl Sep 20, 2024
5bf8b35
Merge branch 'main' into set-default-risk-in-model
originalsouth Sep 23, 2024
967d41b
Merge branch 'main' into set-default-risk-in-model
underdarknl Sep 23, 2024
d30b33f
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Sep 23, 2024
86fe7d5
Merge branch 'fix/prevent_race_conditions_between_event_manager_and_s…
originalsouth Sep 23, 2024
dca2b20
Merge branch 'set-default-risk-in-model' into feature/nibbles
originalsouth Sep 23, 2024
7699d93
Fixes for idle run
originalsouth Sep 23, 2024
0eb106f
Merge branch 'main' into feature/nibbles
originalsouth Sep 24, 2024
2ed89fb
Manual merge
originalsouth Oct 14, 2024
d9c9fa2
Revert "Set default findingtype risk in model instead of in bit (#3562)"
originalsouth Oct 14, 2024
20c5abf
Pre-commit after revert
originalsouth Oct 14, 2024
2d09141
Remove bogus rlu_cache
originalsouth Oct 15, 2024
6adeffe
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 16, 2024
f3f4277
Register origins and add parameters begins
originalsouth Oct 16, 2024
ef9ad80
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 16, 2024
5546cd8
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 16, 2024
cf2f04c
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 16, 2024
6fd5f74
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 29, 2024
1b49c3b
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 30, 2024
b28ae84
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 30, 2024
8b0f50d
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Oct 31, 2024
f140e87
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 4, 2024
be03bf8
Add blocklist and ooi reuse to inference
originalsouth Nov 4, 2024
852ec3e
Fix runner
originalsouth Nov 4, 2024
ed4c40a
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 4, 2024
df9a329
Basic nibbler
originalsouth Nov 6, 2024
5908b42
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 6, 2024
2de975d
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 7, 2024
a67b297
Add more boilerplating
originalsouth Nov 7, 2024
f20cb4b
Check clearance for seed OOI in nibbles
originalsouth Nov 7, 2024
d706b35
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 7, 2024
49e1116
Add unit test
originalsouth Nov 7, 2024
8ff6fac
Add unit test
originalsouth Nov 7, 2024
a9da549
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 7, 2024
6fbcf12
Make SonarClaus Happier
originalsouth Nov 7, 2024
bd7b82d
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 12, 2024
13400b3
More testing and fixing
originalsouth Nov 12, 2024
137b687
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 12, 2024
aa66104
Moves towards a new niddles
originalsouth Nov 13, 2024
4d9baa2
Purge NMAX
originalsouth Nov 13, 2024
63cdaec
Another day another design
originalsouth Nov 14, 2024
f337ee3
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 14, 2024
a18929b
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 17, 2024
e35b101
Add multivariable support
originalsouth Nov 18, 2024
0c8a6bb
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 18, 2024
d320be2
Refactor
originalsouth Nov 19, 2024
87909ae
Fix typing
originalsouth Nov 19, 2024
4b853d9
Refactor
originalsouth Nov 19, 2024
d084a38
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 19, 2024
5266ccd
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 20, 2024
e7b3a5a
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 20, 2024
bd59705
Mostly fix nibble-origins -> nibblettes
originalsouth Nov 21, 2024
e9a4576
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 21, 2024
8c6d6e5
Add comment
originalsouth Nov 21, 2024
9890402
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 26, 2024
ac80ae0
Give me the $$$ AWK input
originalsouth Nov 26, 2024
d978519
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 26, 2024
6272afc
Faster serialization
originalsouth Nov 27, 2024
82b6ad4
Skip encoding
originalsouth Nov 27, 2024
dee4a4a
Revert "Faster serialization"
originalsouth Nov 27, 2024
c40537d
nibblette -> nibblet
originalsouth Nov 27, 2024
9e0a0ca
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 27, 2024
d19812d
Test re-evaluation
originalsouth Nov 27, 2024
5e5ff0b
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Nov 27, 2024
cc73cf0
Fix double dict entry "bug"
originalsouth Nov 28, 2024
986b32d
Run all nibbles not touched by nibblets
originalsouth Nov 28, 2024
40f9093
Cleanup code
originalsouth Dec 2, 2024
555dcd5
Manual merge poetry.lock
originalsouth Dec 2, 2024
4bd5198
Rais non-nibblet origins
originalsouth Dec 2, 2024
b0352d6
Better specializations
originalsouth Dec 2, 2024
16e5259
Fix requirements
originalsouth Dec 2, 2024
603c943
Don't change the integration test
originalsouth Dec 2, 2024
c5da166
Add FIXME comment
originalsouth Dec 2, 2024
6751284
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 2, 2024
6dd8b9b
Type fix
originalsouth Dec 2, 2024
0ebdc9d
Add OOI caching
originalsouth Dec 2, 2024
09fd601
Remove useless tests
originalsouth Dec 2, 2024
8cf3533
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 3, 2024
edb8822
Serialization fixes
originalsouth Dec 4, 2024
4d5ef74
Remove parameter hash
originalsouth Dec 4, 2024
193342e
Add first (test) nibblet
originalsouth Dec 4, 2024
665439c
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 4, 2024
715036f
Minor updates
originalsouth Dec 4, 2024
2b246c6
Minimum scanlevel per variable
originalsouth Dec 4, 2024
7ad6614
Remove min scan level requirements (for now)
originalsouth Dec 4, 2024
de0c783
Fixed some patched bit tests
originalsouth Dec 4, 2024
7b2417b
Remove legacy min_scan_level
originalsouth Dec 4, 2024
03eea29
Nibble functionality in router
originalsouth Dec 4, 2024
14c9f0e
Add first real nibble
originalsouth Dec 4, 2024
8185ed1
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 4, 2024
f73ccb1
Fix Octopoes' integration tests
originalsouth Dec 4, 2024
23af315
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 4, 2024
0f0204a
Fix Octopoes' tests
originalsouth Dec 4, 2024
58e4e5f
Redo pad_level
originalsouth Dec 4, 2024
f5b1a78
Migrate 1 to N bits to nibbles
originalsouth Dec 5, 2024
ae1d35c
Fix dov nibble
originalsouth Dec 5, 2024
128be59
Fix SPF header
originalsouth Dec 5, 2024
ea90a6d
website discovery bit to nibble
noamblitz Dec 5, 2024
a857ee3
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 9, 2024
c7e6a34
Implement recalculate for nibbler
originalsouth Dec 9, 2024
d03014a
Implement recalculate for nibbler
originalsouth Dec 9, 2024
1813b23
Fix octopoes tests (non-integration)
originalsouth Dec 9, 2024
d9dac77
Add reset routine
originalsouth Dec 9, 2024
4315910
Allow queries to be callable
originalsouth Dec 10, 2024
cfb5266
Update callable queries
originalsouth Dec 10, 2024
73dddf1
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 10, 2024
c044dd0
Test for callability
originalsouth Dec 10, 2024
f731775
Retire the dollars
originalsouth Dec 10, 2024
04c8cda
add failing parent ooi type test
noamblitz Dec 10, 2024
df49296
check types with isinstance
noamblitz Dec 10, 2024
1495aef
Retire port_common
originalsouth Dec 10, 2024
d3686de
Manual merge of main
originalsouth Dec 10, 2024
887abee
Fix event manager test
originalsouth Dec 10, 2024
8e0c9ee
Remove printer
originalsouth Dec 10, 2024
b224e4c
Remove port_common tests from test
originalsouth Dec 10, 2024
dc81654
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 10, 2024
e703325
Delete Nibblets properly
originalsouth Dec 11, 2024
9d426d6
Better deletion II
originalsouth Dec 11, 2024
653b004
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 11, 2024
9e3ba58
Remove dangling affirmation a bit later
originalsouth Dec 12, 2024
eb5f6ec
Remove redundant nibble definitions in test_nibbles
originalsouth Dec 12, 2024
191c9ae
More utility functions and maintenance
originalsouth Dec 12, 2024
884a3d7
POJO --> BaseModel
originalsouth Dec 12, 2024
fda34e8
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 12, 2024
0daf3a7
Convert check_hsts_header
originalsouth Dec 12, 2024
5701a6c
Convert check_hsts_header II
originalsouth Dec 12, 2024
2109257
Port disallowed_csp_hostnames
originalsouth Dec 12, 2024
2eef723
allow optional nibble params
noamblitz Dec 12, 2024
c88d03b
Port ooi_in_headers
originalsouth Dec 12, 2024
67fe804
Port ooi_in_headers tests
originalsouth Dec 12, 2024
fe40e42
Add referencefield?
originalsouth Dec 12, 2024
a579212
Add referencefield
originalsouth Dec 12, 2024
d0bdaf9
Add referencefield
originalsouth Dec 12, 2024
b49e831
port missing_spf bit with tests
noamblitz Dec 13, 2024
c3d2763
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 13, 2024
03baf16
fix hsts header nibble with tests
noamblitz Dec 13, 2024
3521a97
check for bit id in query
noamblitz Dec 13, 2024
366e830
Fix itests
originalsouth Dec 16, 2024
9ebe56e
Update config ooi nibbles
originalsouth Dec 16, 2024
a1d02f7
Support nibbles with duplicate parameters in signature
originalsouth Dec 16, 2024
66df903
Fix bit_id for disallowed_csp_hostnames nibble
originalsouth Dec 16, 2024
ac699ef
Unit tests for objectify
originalsouth Dec 16, 2024
c51dca9
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 16, 2024
0ecceec
Introduce tests for config nibbles
originalsouth Dec 17, 2024
66a287a
Fix tests for config nibbles
originalsouth Dec 17, 2024
38dc032
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 17, 2024
1e81c09
Bypass SonarCloudSecurity check?
originalsouth Dec 17, 2024
e9b29b4
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 17, 2024
b74d36a
Fix config queries
originalsouth Dec 18, 2024
779afb7
Patch the ROBOT tetst (there should be more objects with the nibble a…
originalsouth Dec 18, 2024
ec0f8bd
Remove nibbles reset routine (unwanted and premature for now)
originalsouth Dec 18, 2024
15304bc
Update disallow_csp_hostnames from upstream
originalsouth Dec 18, 2024
2076e50
Allow none configs in new nibble
originalsouth Dec 18, 2024
d7403c5
Retire perform_writes option
originalsouth Dec 18, 2024
719da7c
Make sonarcloud happier
originalsouth Dec 18, 2024
7524a11
Make sonarcloud happier
originalsouth Dec 18, 2024
a2d383c
Make sonarcloud happier
originalsouth Dec 18, 2024
e680adc
Cleanup unused code
originalsouth Dec 18, 2024
b4181f1
Allow reruns when nibbles are updated (part I)
originalsouth Dec 18, 2024
e27fbbe
Add retrieve functionality
originalsouth Dec 19, 2024
777b1b8
Implement yields
originalsouth Dec 19, 2024
7cd694f
Add update nibble routines
originalsouth Dec 19, 2024
acc8bbc
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 23, 2024
8106a26
Document objectify a bit
originalsouth Dec 23, 2024
e4dbe69
Rename objectify --> parse_as
originalsouth Dec 23, 2024
cb7ebfb
Beginnings of NibbleRepository etc
originalsouth Dec 24, 2024
22b025f
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 24, 2024
0cd92b2
Actually add the NibbleRepository
originalsouth Dec 24, 2024
ee92430
Fix integration tests
originalsouth Dec 24, 2024
358ec81
Fix tests...
originalsouth Dec 24, 2024
f1d8d15
Merge remote-tracking branch 'origin/main' into feature/nibbles
originalsouth Dec 26, 2024
2b92683
Implement centralized Nibble toggle mechanism
originalsouth Dec 26, 2024
8b1f232
Federilized Nibblers
originalsouth Dec 26, 2024
7149279
Federilized Nibblers II
originalsouth Dec 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions octopoes/bits/ask_disallowed_domains/bit.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/ask_port_specification/bit.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/ask_url_params_to_ignore/bit.py

This file was deleted.

9 changes: 0 additions & 9 deletions octopoes/bits/check_cve_2021_41773/bit.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/check_hsts_header/bit.py

This file was deleted.

9 changes: 0 additions & 9 deletions octopoes/bits/cipher_classification/bit.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/default_findingtype_risk/bit.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/disallowed_csp_hostnames/bit.py

This file was deleted.

9 changes: 0 additions & 9 deletions octopoes/bits/domain_owner_verification/bit.py

This file was deleted.

9 changes: 0 additions & 9 deletions octopoes/bits/expiring_certificate/bit.py

This file was deleted.

6 changes: 0 additions & 6 deletions octopoes/bits/missing_certificate/bit.py

This file was deleted.

14 changes: 0 additions & 14 deletions octopoes/bits/missing_spf/bit.py

This file was deleted.

29 changes: 0 additions & 29 deletions octopoes/bits/missing_spf/missing_spf.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/oois_in_headers/bit.py

This file was deleted.

6 changes: 0 additions & 6 deletions octopoes/bits/port_common/bit.py

This file was deleted.

37 changes: 0 additions & 37 deletions octopoes/bits/port_common/port_common.py

This file was deleted.

4 changes: 0 additions & 4 deletions octopoes/bits/spf_discovery/bit.py

This file was deleted.

10 changes: 0 additions & 10 deletions octopoes/bits/url_classification/bit.py

This file was deleted.

15 changes: 0 additions & 15 deletions octopoes/bits/website_discovery/bit.py

This file was deleted.

25 changes: 0 additions & 25 deletions octopoes/bits/website_discovery/website_discovery.py

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
import json
from collections.abc import Iterator
from pathlib import Path
from typing import Any

from octopoes.models import OOI
from octopoes.models.ooi.network import Network
from octopoes.models.ooi.question import Question


def run(input_ooi: Network, additional_oois: list, config: dict[str, Any]) -> Iterator[OOI]:
def nibble(input_ooi: Network) -> Iterator[OOI]:
network = input_ooi

with (Path(__file__).parent / "question_schema.json").open() as f:
Expand Down
4 changes: 4 additions & 0 deletions octopoes/nibbles/ask_disallowed_domains/nibble.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from nibbles.definitions import NibbleDefinition, NibbleParameter
from octopoes.models.ooi.network import Network

NIBBLE = NibbleDefinition(id="ask-disallowed-domains", signature=[NibbleParameter(object_type=Network)])
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
import json
from collections.abc import Iterator
from pathlib import Path
from typing import Any

from octopoes.models import OOI
from octopoes.models.ooi.network import Network
from octopoes.models.ooi.question import Question


def run(input_ooi: Network, additional_oois: list, config: dict[str, Any]) -> Iterator[OOI]:
def nibble(input_ooi: Network) -> Iterator[OOI]:
network = input_ooi

with (Path(__file__).parent / "question_schema.json").open() as f:
Expand Down
4 changes: 4 additions & 0 deletions octopoes/nibbles/ask_port_specification/nibble.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from nibbles.definitions import NibbleDefinition, NibbleParameter
from octopoes.models.ooi.network import Network

NIBBLE = NibbleDefinition(id="ask-port-specification", signature=[NibbleParameter(object_type=Network)])
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
import json
from collections.abc import Iterator
from pathlib import Path
from typing import Any

from octopoes.models import OOI
from octopoes.models.ooi.network import Network
from octopoes.models.ooi.question import Question


def run(input_ooi: Network, additional_oois: list, config: dict[str, Any]) -> Iterator[OOI]:
def nibble(input_ooi: Network) -> Iterator[OOI]:
network = input_ooi

with (Path(__file__).parent / "question_schema.json").open() as f:
Expand Down
4 changes: 4 additions & 0 deletions octopoes/nibbles/ask_url_params_to_ignore/nibble.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from nibbles.definitions import NibbleDefinition, NibbleParameter
from octopoes.models.ooi.network import Network

NIBBLE = NibbleDefinition(id="ask_url_params_to_ignore", signature=[NibbleParameter(object_type=Network)])
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
from collections.abc import Iterator
from typing import Any

from octopoes.models import OOI
from octopoes.models.ooi.findings import CVEFindingType, Finding
from octopoes.models.ooi.web import HTTPHeader


def run(input_ooi: HTTPHeader, additional_oois: list, config: dict[str, Any]) -> Iterator[OOI]:
def nibble(input_ooi: HTTPHeader) -> Iterator[OOI]:
header = input_ooi
if header.key.lower() != "server":
return
Expand Down
4 changes: 4 additions & 0 deletions octopoes/nibbles/check_cve_2021_41773/nibble.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from nibbles.definitions import NibbleDefinition, NibbleParameter
from octopoes.models.types import HTTPHeader

NIBBLE = NibbleDefinition(id="check_cve_2021_41773", signature=[NibbleParameter(object_type=HTTPHeader)])
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
import datetime
from collections.abc import Iterator
from typing import Any

from octopoes.models import OOI, Reference
from octopoes.models.ooi.config import Config
from octopoes.models.ooi.findings import Finding, KATFindingType
from octopoes.models.ooi.web import HTTPHeader


def run(input_ooi: HTTPHeader, additional_oois: list, config: dict[str, Any]) -> Iterator[OOI]:
def nibble(input_ooi: HTTPHeader, config: Config | None) -> Iterator[OOI]:
header = input_ooi
if header.key.lower() != "strict-transport-security":
return

one_year = datetime.timedelta(days=365).total_seconds()

max_age = int(config.get("max-age", one_year)) if config else one_year
max_age = int(str(config.config.get("max-age", one_year))) if config and config.config else one_year
findings: list[str] = []

headervalue = header.value.lower()
Expand Down
Loading
Loading