Migrate to Official Neo4j Driver Sessions (remove py2neo)

[stryker2k2]

Description:

• Updated our neo4j output to use the official NEO4J Driver Sessions instead of the deprecated py2neo project.

Ticket/Issue

• Update Neo4j Library #778

To Review:

• Run BBOT Module (using -om neo4j) on a website that multiple modules would emit on
  • bbot -t blacklanternsecurity.com -f subdomain-enum -rf passive -om neo4j
  • Expect a nice looking neo4j graph
  • Expect 0 Unrelated Nodes in neo4j graph
  • Expect 0 Errors from "neo4j.py:handle_batch()"
  • Expect 0 Unrelated Nodes in neo4j graph
• Code Review
• Tests Review

Notes:

• You can uncomment out lines in /db/neo4j.py:__init__ to force the deletion of all previous records stored in the neo4j database before commiting new nodes & relationships to neo4j.

[stryker2k2]

Github Tests failing. Investigating now.
FAILED bbot/test/test_step_1/test_modules_basic.py::test_modules_basic - ModuleNotFoundError: No module named 'neo4j'

[TheTechromancer]

For reference, py2neo's queries:

// merge event
UNWIND $data AS r
MERGE (_:SCAN {id:r['id']})
SET _ += r
SET _:SCAN
RETURN id(_)
// {'data': [{'id': 'SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0', 'data': 'grievous_ross (SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0)', 'scope_distance': 0, 'scan': 'SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0', 'source': 'SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0', 'tags': ['in-scope'], 'module_sequence': 'TARGET'}]})

// merge relationship
UNWIND $data AS r
MATCH (a) WHERE id(a) = r[0]
MATCH (b) WHERE id(b) = r[2]
MERGE (a)-[_:TARGET]->(b)
SET _ += r[1]
RETURN id(_)
// {'data': [[7, {'timestamp': datetime.datetime(2024, 3, 21, 20, 49, 8, 68850)}, 7]]})

[stryker2k2]

For reference, py2neo's queries:

// merge event
UNWIND $data AS r
MERGE (_:SCAN {id:r['id']})
SET _ += r
SET _:SCAN
RETURN id(_)
// {'data': [{'id': 'SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0', 'data': 'grievous_ross (SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0)', 'scope_distance': 0, 'scan': 'SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0', 'source': 'SCAN:2a510ba2490ca8e50c2a681a64068e785275fdd0', 'tags': ['in-scope'], 'module_sequence': 'TARGET'}]})

// merge relationship
UNWIND $data AS r
MATCH (a) WHERE id(a) = r[0]
MATCH (b) WHERE id(b) = r[2]
MERGE (a)-[_:TARGET]->(b)
SET _ += r[1]
RETURN id(_)
// {'data': [[7, {'timestamp': datetime.datetime(2024, 3, 21, 20, 49, 8, 68850)}, 7]]})

Sweet. Thanks! It is in-line with what I have. It seems that py2neo and I looked at the same Neo4j Documentation.

Make Node
MERGE (:{event_type} " + "{"{event}: {event_json.get(str(item))}"})

Make Relationship
MATCH (source:{source_type} {{id: "{source_id}"}})
MATCH (target:{dest_type} {{id: "{dest_id}"}})
MERGE (source)-[r:{relation_type}]->(target)
RETURN COUNT(source) as total

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 97.43590% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 92%. Comparing base (ca10906) to head (1eb4f35).
Report is 9 commits behind head on dev.

Files	Patch %	Lines
bbot/modules/output/neo4j.py	96%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #1205   +/-   ##
=====================================
- Coverage     92%     92%   -0%     
=====================================
  Files        312     313    +1     
  Lines      19825   19907   +82     
=====================================
+ Hits       18215   18280   +65     
- Misses      1610    1627   +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.