CI specimin integration #330
Conversation
| - name: Debug - Show Current Branch | ||
| run: | | ||
| set -ex | ||
| echo "Current branch: $(git branch --show-current)" |
There was a problem hiding this comment.
This step (and other debug steps) should be removed before we merge this script.
| - name: Clone ASHE Project | ||
| run: | | ||
| set -ex | ||
| git clone https://github.com/njit-jerse/ASHE_Automated-Software-Hardening-for-Entrypoints ASHE |
There was a problem hiding this comment.
Rather than cloning the ASHE repo directly, this should use the git-clone-related program available here: https://github.com/plume-lib/git-scripts/blob/main/git-clone-related instead. That program permits CI to work correctly when a developer needs to make changes that must be coordinated across multiple repositories.
| set -ex | ||
| git clone https://github.com/njit-jerse/ASHE_Automated-Software-Hardening-for-Entrypoints ASHE | ||
|
|
||
| - name: Create ASHE Clone SPACE Directory |
There was a problem hiding this comment.
What is the purpose of this "Clone SPACE Directory"? It seems superfluous to me.
| chmod +w ASHE/src/main/resources/config.properties | ||
| ls -l ASHE/src/main/resources/config.properties | ||
|
|
||
| - name: Update ASHE Config File to update SPECIMIN path |
There was a problem hiding this comment.
"Specimin" is a name, not an acronym. So, it should not be styled as "SPECIMIN", since it doesn't stand for anything.
| set -ex | ||
| current_accuracy=$(cat current_run_accuracy_percentage.txt) | ||
| echo "Current accuracy: $current_accuracy" | ||
| previous_run_accuracy=${{ secrets.LATEST_SPECIMIN_EVAL_PERCENTAGE }} |
There was a problem hiding this comment.
I'm concerned that this is a single global variable for all pull requests. Consider the following scenario in the future:
- on the
mainbranch, the last CI run produced 90% accuracy - from the
feature-1branch, developer A opens a PR with 85% accuracy. This CI task fails, because 85 < 90 (all good so far!) - from the
feature-2branch, developer B opens another PR with 87% accuracy. Will this CI task pass? I think it will, because the latest run will be the run offeature-1that failed with 85% accuracy, and 87 > 85.
So, I think that this really needs to only be updated if the current branch is main: otherwise, there could be confusion between PRs.
This change will run a workflow to run accuracy check for changes made in specimin via a pull request or commit on main branch of project specimin