scripts to create several SAST reports #158
Conversation
|
Wow! Thanks for these. I'll test them out. |
| page=$((page+1)) | ||
| done | ||
|
|
||
| echo "$result" > "$result_file" |
There was a problem hiding this comment.
When I run this, this happens: Dave.Wichers@USC02FJ2VXMD6T Benchmark % scripts/runSonarQube.sh
[INFO] BuildTimeEventSpy is registered.
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/sonar-maven-plugin/maven-metadata.xml
[WARNING] Could not transfer metadata org.codehaus.mojo:sonar-maven-plugin/maven-metadata.xml from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/org/codehaus/mojo/sonar-maven-plugin/maven-metadata.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.859 s
[INFO] Finished at: 2021-09-14T14:38:43-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Error resolving version for plugin 'org.codehaus.mojo:sonar-maven-plugin' from the repositories [local (/Users/Dave.Wichers/.m2/repository), central (https://repo.maven.apache.org/maven2)]: Plugin not found in any plugin repository -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginVersionResolutionException
^C
How is this supposed to work?
There was a problem hiding this comment.
This is a copy-paste-command from SonarQube. Did you update the credentials in the file?
There was a problem hiding this comment.
If credentials need to be changed in the script, then the script should come with NO credentials, detect when they have not been provided, and provide instructions on what to do. Look at the runBenchmark_wContrast.sh as an example of this. It's probably better if the credentials sit in a separate file (again like Contrast), so user's can edit the credential file separately. This allows them to pick up improvements to the script via git pull, without merge conflicts.
There was a problem hiding this comment.
done. please check :-)
There was a problem hiding this comment.
is there a SaaS instance of SonarQube we can use? If so, we should 'suggest' those in the SonarQubeCredential file. I'm not super interested in setting up a local SonarQube instance as that's a pain. If a user can use a SaaS instance, and simply sign up for an account to use first, that would make it really easy for them to scan Benchmark w/SonarQube.
There was a problem hiding this comment.
I just used https://docs.sonarqube.org/latest/setup/get-started-2-minutes/ without any configuration. Just wanted to have a running script.
There was a problem hiding this comment.
That's great. Given that it is using docker, can you update your script to launch the Docker VM automatically, and then interact with it. Then you could embed the default credentials right into it. Also, you could put instructions on how to change the script to use your own SonarQube instance if you don't want to use Docker. Maybe create a separate SonarQubeUsageInstructions.txt file that explains its default behavior, and how to make these changes if you want to.
I guess that might be trickier than I thought. Is there any way to automate most of this? If not, we should put info in SonarQubeUsageInstructions.txt that explains things like:
- Download/run the docker image with this command (X)
- Login via your browser at URL and change your password.
- Click on manual setup.
- Create a project named Benchmark, with a key,
- etc.
Once all this is done, enter the credentials into: SonarQubeCredentials.sh
Then run scripts/runSonarQube.sh
Also - I noticed that one of the dependencies requires Java 11. We are trying to stick with Java 8 (if possible). Is it possible to downgrade to only Java 8 components? If not, we should check the Java version and if prior to 11, stop the script and tell them Java 11+ is required to run this script.
| -v /var/run/docker.sock:/var/run/docker.sock \ | ||
| -v "$(pwd)":/src horuszup/horusec-cli \ | ||
| horusec start -p /src -P "$(pwd)" -t 3600 \ | ||
| -o="json" -O="$result_file" |
There was a problem hiding this comment.
When I run this, this is what I get:
Dave.Wichers@USC02FJ2VXMD6T Benchmark % scripts/runHorusec.sh
time="2021-09-16T18:57:03Z" level=info msg="Set log file to /tmp/horusec/horusec-log-2021-09-16 18:57:03.log"
time="2021-09-16T18:57:23Z" level=warning msg="{HORUSEC_CLI} When starting the analysis WE SKIP A TOTAL OF 280 FILES that are not considered to be analyzed. To see more details use flag --log-level=debug"
time="2021-09-16T19:00:20Z" level=warning msg="{HORUSEC_CLI} PLEASE DON'T REMOVE ".horusec" FOLDER BEFORE THE ANALYSIS FINISH! Don’t worry, we’ll remove it after the analysis ends automatically! Project sent to folder in location: [/src/.horusec/ac920f08-6ad4-4096-9657-abaf04f9d702]"
time="2021-09-16T19:00:20Z" level=info msg="Hold on! Horusec is still analyzing your code. Timeout in: 3600s"
...
time="2021-09-16T19:05:50Z" level=info msg="Hold on! Horusec is still analyzing your code. Timeout in: 3270s"
And after it stopped, I looked for a results file in /results, and nothing new was there. Where does the result file go? Is it left on the Docker image and not exported to the local /results folder?
There was a problem hiding this comment.
What OS are you using? Just ran the script and the result file appeared in the results folder. (macOS 11.x)
There was a problem hiding this comment.
I'm on MacOS too. This time when I ran it, this happened: Benchmark % scripts/runHorusec.sh
time="2021-09-18T15:31:08Z" level=info msg="Set log file to /tmp/horusec/horusec-log-2021-09-18 15:31:08.log"
time="2021-09-18T15:31:49Z" level=warning msg="{HORUSEC_CLI} When starting the analysis WE SKIP A TOTAL OF 32103 FILES that are not considered to be analyzed. To see more details use flag --log-level=debug"
time="2021-09-18T15:34:11Z" level=warning msg="{HORUSEC_CLI} PLEASE DON'T REMOVE ".horusec" FOLDER BEFORE THE ANALYSIS FINISH! Don’t worry, we’ll remove it after the analysis ends automatically! Project sent to folder in location: [/src/.horusec/1a7af08d-2a75-4404-9fcb-550d633fdafc]"
time="2021-09-18T15:34:11Z" level=info msg="Hold on! Horusec is still analyzing your code. Timeout in: 3600s"
ERRO[0197] error waiting for container: invalid character 'u' looking for beginning of value
I'm using Docker Desktop 3.6.0 w/Engine 20.10.0
There was a problem hiding this comment.
I upgraded to Docker Desktop 4.0.1 which has Engine 20.10.8 (I might have actually had Engine 20.10.8 before actually, not sure). Anyway, that didn't help.
There was a problem hiding this comment.
error waiting for container: invalid character 'u' looking for beginning of value
This sounds like a docker/mac issue. At least according to this:
Can you check this on another machine?
There was a problem hiding this comment.
I just pulled your changes and tried again and now get this?? Benchmark % scripts/runHorusec.sh
docker: Error response from daemon: Bad response from Docker engine.
See 'docker run --help'.
There was a problem hiding this comment.
So, I rebooted my Mac, and tried again. This time it ran but still nothing in /results. However, I did find this: .horusec/a4fa4651-3281-43c4-87f2-47f45488ae75/result.json
And inside .horusec/a4fa4651-3281-43c4-87f2-47f45488ae75/. is a copy of ALL the files from the Benchmark project. Is that the result file that is supposed to be moved to /results, and renamed of course?
There was a problem hiding this comment.
.horusec (and all within) should only exist temporary and be deleted after the run. Yes, this is a complete copy (comes from Horusec, not me!). There shouldn't be a need to manually move files since the flag -O (in combination with -o) should create the file in /results...
Source: https://horusec.io/docs/cli/resources/#3-flags
I updated my docker, did mvn clean, deleted the image and pulled it again - but still it doen't fail and get the expected file in the expected directory. I'll try this on my work computer tomorrow (mac, too). Getting out of ideas though... 😅
There was a problem hiding this comment.
Added --rm to the script. Don't think this will make a difference but still it's cleaner (haha...)
This reverts commit 70e7c48.
| || "Make sure creating this cookie without the \"secure\" flag is safe here." | ||
| .equals(message) | ||
| || "Make sure that hashing data is safe here.".equals(message) | ||
| || "Make sure this weak hash algorithm is not used in a sensitive context here." | ||
| .equals(message) | ||
| || "Make sure creating this cookie without the \"HttpOnly\" flag is safe." | ||
| .equals(message))) { | ||
| System.out.println( |
There was a problem hiding this comment.
I'm noticing that the old results from 5+ years ago: https://github.com/OWASP-Benchmark/BenchmarkJava/blob/master/results/Benchmark_1.2-sonar-Java-Plugin-v3.14-330.xml. Are far better than what I can generate today with the script you just provided and these updates to the SonarQubeJsonReader. Any clue why? Doesn't seem right to me.
There was a problem hiding this comment.
I also scored the Benchmark_1.2-sonarqube-v9.1.0.47736.json results I just generated and the scores aren't any different with or without these code changes to this Parser. Are the changes you made supposed to improve the score? The ONLY score I'm seeing is: Weak Encryption Algorithm -100.00% 23.28% 76.72%, and all zeroes for every other category. Is that what you are seeing too?
Whereas with the OLD Java/xml version (5 years ago), we saw: 100% for Insecure Cookie, Encryption, and Randomness, 69% for hashing, 100-100 for LDAP, and 83-85 for CMDi. Where did those rules go? Are they off by default and we need to figure out how to turn them back on? Or did they move and we aren't recognizing them? Just look at the default scorecard that comes with Benchmark to see those old results, or include the Benchmark_1.2-sonar-Java-Plugin-v3.14-330.xml results file in the new scorecard you generate to compare them side-by-side.
|
I'm going to merge this 'as is' so far. I still would like to see some tweaks/improvements to: runHorusec.sh (as it still doesn't work for me), and runSonarQube.sh per my last set of comments. Let's work on this changes in the new OWASP-Benchmark/BenchmarkUtils project, after I make it public. |
As promised scripts to generate report files, auto-fetching benchmark and tool (if available) versions.
This also contains generation of SonarQube report using their api. Since I combined hotspots and issues (because both categories contain vulnerabilities for whatever reason) I also updated the SonarQubeJsonReader. This might also solve #33 and #117.