Presets

[TheTechromancer]

This PR contains the new Presets feature, as mentioned here.

TODO:

• Consolidate bbot initialization into CORE
• Create Preset class
  • Integrate with CORE
  • Integrate with Scanner
    • Remove dependency on Scanner (so Scanner needs Preset, but not vice versa)
      • This will require a little reorganization, since currently some of the config-aware helpers rely on scan to emit events (which must be associated with the main scan).
        • Preset->Target
        • Cloud helper
        • DNS helper
        • Scan manager (for DNS children)
        • Scan (for root event)
  • Integrate with CLI (arg parsing)
• Evict flag/module resolution logic from cli.py
  • Find a new home for it
• Implement module dependencies
  • Make flag/module resolution recursive
• Make module loader independent of presets (there should be only one instance of it, not one per preset)
  • Implement custom module directories
  • Ensure each module file is read only once (import directly from preloaded string)
    • This was a nice idea, but it ended up being infeasible because of the complexities of module inheritance and the need modules to be topologicaliclly sorted by dependency before importing. Sticking to the original system; it works fine.
• Reimplement config environment variables (DEBUG, etc.)
  • Reimplement module config find-and-replace for #{custom_variables}
• Cache preloaded modules (for faster startup)
• Implement Preset YAML save/load
  • Implement Preset merge functionality (enable multiple presets)
    • conflict awareness/handling
    • recursive preset inclusion (presets that include other presets)
  • enable preset loading from multiple directories (PRESET_PATH)
    • tolerance of .yaml/.yml confusion, no extension
• Ability to list presets + their descriptions
• Output modules - reasonable defaults, ability to override
• Move config find-and-replace functionality to Preset.bake()
• Separate Preset.modules for cleaner+reversible serialization
• Symlink code presets to ~/.bbot/presets
• Save scan's preset to its output folder
• Ensure config files are created if not present
  • retire secrets.yml
• Suite of default presets
  • subdomain enum
  • web spider
  • dirbust
  • cloud enum
  • email enum
  • code enum
  • web basic
  • web advanced
• Restore cli log level toggling
• Tests:
  • Get failing tests to pass
    • inconsistency in bbot/test/test_step_1/test_manager_deduplication.py:95
  • CORE:
    • config merging
    • default / custom config split
  • Presets:
    • ensure every preset has a description
    • preset save<-->load
      • recursive inclusion
        • circular inclusions
    • require flags / exclude flags / exclude modules
      • (including multiple at a time)
      • (including internal and output modules (exclude module only))
    • preset merging
      • log verbosity levels
      • modules / flags
        • custom module directories
          • proper validation/sanitization of module names, flags, custom config options
    • disablement of internal modules in the config
    • CLI
      • -p / -c (files)
      • -p / -c (dot syntax)
      • make sure we have tests for every CLI option
      • deps behavior
  • stdout output module
• Documentation:
  • user manual
  • developer docs
    • code comments / docstrings
  • readme tlc
  • purge references to secrets.yml

[stryker2k2]

I have reviewed this branch and it looks good.

During this review, I launched BBOT and tested every code snippet found in the documentation and checked for errors. I reviewed the Preset TODO list to ensure that every change was indeed found in the code and completed. I reviewed every user document (README, etc), read every line, and followed every link in the documentation to ensure their functionality.

During the pytests review, I ran all the tests with debug logs viewable and watched for any false-positives. I paid close attention to the tests which had the most changes.

During the code review, I paid extra close attention to presets.py, scanner.py, environ.py, files.py, depsinstaller.py. I asked questions directly to the author on code snippets that I didn't understand and asked for feedback when I didn't see errors that I expected to see.

I have been using the presets branch while it was being developed. Thus, during this review, I focused on finding bugs that I had seen before. Every bug that I had encountered in the past is no where to be seen now.

I am submitting my approval of this branch and recommend it to be merged.