Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Module: GitHub workflow logs #1335

Merged

Conversation

domwhewell-sage
Copy link
Contributor

This PR adds a new module to download workflow logs from a repository as mentioned in #1305.

It will always try all workflows in the repository and by default 1 successful log is downloaded for each and you can specify num_logs up to a maximum of 100 logs for each workflow.

It raises FILESYSTEM events for the downloaded workflow logs archive.

The plan is to run trufflehog against these archives but first I want to double check trufflehog runs against them without loads of duplicates (Unzipping the archive manually there's a large logfile and smaller logfile "chunks" that seem to duplicate the content of the largelog)

bbot -t blacklanternsecurity.com -m github_org, github_workflows --config modules.github_org.api_key=<api_token>

@domwhewell-sage
Copy link
Contributor Author

Marking this for review now, unfortunately trufflehog is duplicating the discovered keys as technically they are within 2 different "lines" of the zip file probably something to think about in our trufflehog module if we could do an internal set() to de-dupe discovered secrets.

Also the repo I used to test this on had an aws_access_key, aws_secret_access_key and aws_session_token in the workflow log and trufflehog wasn't picking it up so that's a bug I'll have to pickup with the developers of that tool.

Finally the "location" a discovered secret would be found is a run_XXXXXXX.zip file which obviously will mean nothing to the user of bbot so we would need some way of linking this to the original CODE_REPOSITORY event. (#1319 ?)
Theoretically: CODE_REPOSITORY -> FILESYSTEM -> FINDING.

Nothing to change in this module but all things to think about for the trufflehog module changes required to make this module yield secrets

@domwhewell-sage domwhewell-sage marked this pull request as ready for review May 1, 2024 09:56
@TheTechromancer
Copy link
Collaborator

TheTechromancer commented May 1, 2024

For now can we add a description to the FILESYSTEM event that says something like, "these are logs from the GitHub workflow <workflow> on <repo> at <time>"?

@TheTechromancer
Copy link
Collaborator

Nice work on this! I made a small tweak to the error handling, let me know if it looks good.

@domwhewell-sage
Copy link
Contributor Author

I've made a modification to prevent the duplication as the downloaded zip archive contains a structure like

allsteps.txt
folder/
  - step1.txt
  - step2.txt

Therefore a secret could be in allsteps.txt and step2.txt which would make trufflehog raise the finding 2x for the same secret

@TheTechromancer TheTechromancer merged commit eb9c4bc into blacklanternsecurity:dev May 11, 2024
8 checks passed
@domwhewell-sage domwhewell-sage deleted the github_workflow_logs branch May 12, 2024 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants