BUG: Boefje WebpageAnalysis reports absence of HSTS on wrong web asset in report

[okoeroo]

The WebpageAnalysis boefje reports the the absence of HSTS on the wrong asset. This is likely caused by the way python requests (and session) work by default. The default behaviour is to follow a Location header before getting the header data and run a report.

To reproduce

1. Enable WebpageAnalysis
2. Scan a resource where the first URI provides an HSTS header, but sets a Location to an asset without an HSTS header. In this case, where the security.txt is hosted.
3. Run a report.
4. Finding observed: KAT-HSTS-VULNERABILITIES @ Strict-Transport-Security @ https://dopingautoriteit.nl:443/security.txt @ 136.144.185.174

Example site:

• Going to "https://dopingautoriteit.nl:443/security.txt" provides an HSTS header and a Location to "https://www.dopingautoriteit.nl:443/security.txt"
• "https://dopingautoriteit.nl:443/security.txt" does not feature an HSTS header.

Expected behavior
I expected the finding to be on the asset "https://www.dopingautoriteit.nl:443/security.txt" in the report.

OpenKAT version
Branch main
Commit 945ba6e
Time of checkout: Dec 21, 2024

Also:
Tag: v1.17.0

[underdarknl]

In my tests I see the following HSTS finding for "https://dopingautoriteit.nl:443/security.txt"

• The HSTS should include subdomains.

And the following for "https://www.dopingautoriteit.nl:443/security.txt" to which we where redirected.

• The website does not use HTTP Strict Transport Security (HSTS). HSTS ensures that browsers can only access the website using encryption (HTTPS).

When also enabling the security_txt boefje, we see the content of the url https://www.dopingautoriteit.nl/.well-known/security.txt file being ingested, where it should correctly identify the 404 being returned.
This PR should handle the 404 cases:
#2556