Explicit HTTPS scope item results in HTTP equivilant as a the only resulting URL

[harrisonSoS]

Out of the box BBOT other than the following modifications to bbot.yml:

modules:
  nuclei:
    templates: '/home/REDACTED/nuclei-templates/http/cves/REDACTED/CVE-REDACTED.yaml'
    directory_only: false

Using command line:

bbot -f web-basic -m nuclei --allow-deadly -t "https://REDACTED" -v -d

Seemingly, during web-basic modules, the http://REDACTED endpoint equivalent is found and is set as the preferred endpoint as it is the only endpoint sent to nuclei (based on adding -debug to the nuclei command run in bbot/modules/deadly/nuclei.py and viewing the HTTP requests/responses).

The nuclei template in question checks for status 200 and a REGEX on the response. However, hitting http://REDACTED/path/nuclei/template/hits returns a 302 to go to https://REDACTED/path/nuclei/template/hits but this redirect is not followed, the template completes having not satisfied the criteria, and the known vulnerability is not reported. (Vuln is found running nuclei naked with the https:// scope outside of bbot).

Forgive my ignorance, but there doesn't appear to be a flag to alter this behaviour to either a) prefer HTTPS, b) run nuclei on http and https or c) follow nuclei redirects. http_max_redirects has no effect.

Is it possible to get a new config option, default behaviour or workaround for this issue?

Many thanks!

[TheTechromancer]

@harrisonSoS thanks for the report. This sounds like it might be an interesting bug.

There's one way that BBOT works differently from raw nuclei, and that's that the nuclei BBOT module accepts the URLdata type, rather than accepting hosts directly. URLs are generated by the httpx module, which means it's actually up to httpx to find both the http and https versions of the site, and send those to nuclei as separate targets.

I'm curious, what URLs do you have in your output? Do both the HTTP and HTTPS versions show up? Does it behave differently if you specify a host as a target instead of a URL?

[harrisonSoS]

Ran the same BBOT instance with the above config with three different targets: http, https and raw endpoint resulting in vuln found, not found and found respectively:

http://REDACTED - Nuclei vuln located

bbot -f web-basic -m nuclei --allow-deadly -t "http://REDACTED" -v -d
egrep 'URL|VULN' .bbot/scans/debug_http/output.txt

[URL]               	http://REDACTED/	TARGET->httpx	(dir, distance-0, in-scope, ip-REDACTED, status-302)
[URL]               	https://REDACTED/	httpx	(dir, distance-0, http-title-404-not-found, in-scope, ip-REDACTED, status-404)
[VULNERABILITY]     	{"description": "template: [CVE-REDACTED], name: [word-1]", "host": "REDACTED", "severity": "CRITICAL", "url": "https://REDACTED/nuclei/template/endpoint"}	nuclei	(critical, distance-0, in-scope)
[VULNERABILITY]     	{"description": "template: [CVE-REDACTED], name: [word-2]", "host": "REDACTED", "severity": "CRITICAL", "url": "https://REDACTED/nuclei/template/endpoint"}	nuclei	(critical, distance-0, in-scope)
[VULNERABILITY]     	{"description": "template: [CVE-REDACTED], name: [status-3]", "host": "REDACTED", "severity": "CRITICAL", "url": "https://REDACTED/nuclei/template/endpoint"}	nuclei	(critical, distance-0, in-scope)

https://REDACTED - Nuclei vuln NOT located

bbot -f web-basic -m nuclei --allow-deadly -t "https://REDACTED" -v -d
egrep 'URL|VULN' .bbot/scans/debug_https/output.txt

[URL]               	http://REDACTED/	httpx	(dir, distance-0, in-scope, ip-REDACTED, status-302)

REDACTED - Nuclei vuln located

bbot -f web-basic -m nuclei --allow-deadly -t "REDACTED" -v -d
egrep 'URL|VULN' .bbot/scans/debug_no_protocol/output.txt

[URL]               	http://REDACTED/	TARGET->httpx	(dir, distance-0, in-scope, ip-REDACTED, status-302)
[URL]               	https://REDACTED/	httpx	(dir, distance-0, http-title-404-not-found, in-scope, ip-REDACTED, status-404)
[VULNERABILITY]     	{"description": "template: [CVE-REDACTED], name: [word-1]", "host": "REDACTED", "severity": "CRITICAL", "url": "https://REDACTED/nuclei/template/endpoint"}	nuclei	(critical, distance-0, in-scope)
[VULNERABILITY]     	{"description": "template: [CVE-REDACTED], name: [word-2]", "host": "REDACTED", "severity": "CRITICAL", "url": "https://REDACTED/nuclei/template/endpoint"}	nuclei	(critical, distance-0, in-scope)
[VULNERABILITY]     	{"description": "template: [CVE-REDACTED], name: [status-3]", "host": "REDACTED", "severity": "CRITICAL", "url": "https://REDACTED/nuclei/template/endpoint"}	nuclei	(critical, distance-0, in-scope)

[TheTechromancer]

Okay, this is good data. I'll see if I can reproduce it.

[harrisonSoS]

Seemingly this relates to prior URL discovery and httpx rather than it being nuclei related - adjusted issue title to suit.

[TheTechromancer]

Okay, I've been doing some testing and I can't get the bug to trigger.

Can you provide redacted output/debug logs? I'm especially curious if there's anything wacky in the HTTP_RESPONSE. To include those in the output, you can do:

bbot -t target.com -m httpx -c omit_event_types=[]

[harrisonSoS]

You just pointed me in the right direction. httpx is setup to drop 404s: https://github.com/blacklanternsecurity/bbot/blob/stable/bbot/modules/httpx.py#L131

Commenting out:

            # discard 404s from unverified URLs
            if source_event.type == "URL_UNVERIFIED" and status_code in (404,):
                self.debug(f'Discarding 404 from "{url}"')
                continue

in bbot/modules/httpx.py correctly outputs both [URL]'s and finds the nuclei vulnerability.

That's because my http:// endpoint 302's to the https:// endpoint. 302 is not 404, so it's saved as a valid URL. The https:// endpoint throws a 404 so it's scrapped and never saved as a URL.

However, whilst the root / 404's, there are sub pages that 200 a-ok. Which is a fringe but not what I would think is super rare use-case where webservers only respond on a certain path, and if we only hit the root /, it 404s and we don't probe it again, we're missing a vast attack surface.

Do you think it would make sense to either change the default behaviour or have a flag to ignore # discard 404s from unverified URLs in the httpx.py module? --assume-404-alive for example.

What's odd to me, that perhaps makes sense to you, is that 404 check in httpx.py didn't apply on the raw endpoint scope test case above. Perhaps speculate assumes 80 and 443 and stores them as URL's before httpx gets to them?

[TheTechromancer]

Ohh interesting.

Do you think it would make sense to either change the default behaviour or have a flag to ignore # discard 404s from unverified URLs in the httpx.py module? --assume-404-alive for example.

I agree with this. I think making an exception for the root URL would make sense. @liquidsec what do you think?

Perhaps speculate assumes 80 and 443 and stores them as URL's before httpx gets to them?

Yes, speculate will always assume 80 and 443 if no port scanner is enabled.

[TheTechromancer]

This has been fixed in #808. A test has also been written to ensure the correct behavior.

[liquidsec]

@TheTechromancer I agree that a 404 on the root page should be an exception. That is very much "alive" in every scenario I can think of.