Run Specimin on all methods in a project in CI #323
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kelloggm
requested changes
Jul 16, 2024
There was a problem hiding this comment.
Looking through the logs of the CI run, this doesn't appear to be quite working as intended:
- the resulting accuracy is 0%, which is not what we expect
- the cause might be that the logs indicate that Specimin is failing to run with some kind of problem related to the auto-formatter Spotless. Why is that happening?
- the logic uses a deprecated mechanism for storing results
set-update. The logs have instructions for how to use the new, preferred mechanism ("environment files"), which you should look into.
.github/workflows/gradle.yml
Outdated
| @@ -71,7 +71,7 @@ jobs: | |||
| uses: actions/setup-java@v4 | |||
| with: | |||
| distribution: 'corretto' | |||
| java-version: '17' | |||
| java-version: '21' | |||
There was a problem hiding this comment.
What's the reason for changing the Java version used for the regular CI runs? I don't think the Java version here should have any impact on the new CI job that you've added.
Also, why is the Java version important? Does something fail if we use Java 17? If so, that should be fixed: Specimin should run under Java 17+.
| @@ -0,0 +1,66 @@ | |||
| name: specimin-evaluation_CI_with_Gradle | |||
There was a problem hiding this comment.
Why are there two near-copies of this file? Only one of them seems to be running (there is only one new CI job); which is it?
| echo "Contents of Specimin_Eval_Project_List.csv:" | ||
| cat Specimin_Eval_Project_List.csv | ||
|
|
||
| - name: Display CSV File Contents |
There was a problem hiding this comment.
Why are there two steps to display the contents of this CSV file?
| - name: List Files for Debugging | ||
| run: ls -R | ||
|
|
||
| - name: Clone Project B |
|
|
||
| - name: Clone Project B | ||
| run: | | ||
| git clone https://github.com/NiharikaJamble/ASHE_Automated-Software-Hardening-for-Entrypoints ASHE |
There was a problem hiding this comment.
Specimin's CI should, if possible, only reference other projects that are under the njit-jerse organization. Is it necessary to use your fork, or could you use this fork instead?
|
@NiharikaJamble is this ready to review again? If so, please either post a comment that says so or use the GitHub UI to "request another review". |
|
Hi Professor
I am working on two comments that you mentioned. Rest of them are fixed.I
have the following issues/questions regarding the same.
1. Looking through the logs of the CI run, this doesn't appear to be quite
working as intended:
- the resulting accuracy is 0%, which is not what we expect
- the cause might be that the logs indicate that Specimin is failing to
run with some kind of problem related to the auto-formatter Spotless. Why
is that happening?
- the logic uses a deprecated mechanism for storing results set-update.
The logs have instructions for how to use the new, preferred mechanism
("environment files"), which you should look into.
> the latest working log I had shows 90.07%, Can i get the log for
which you encountered this issue?
2. In .github/workflows/gradle.yml
<https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*discussion_r1679630354__;Iw!!DLa72PTfQgg!LGE3Duu6fOGQFBMz6Fo6YAQPluvpcLEbEIlYqZlBSOvLLgyRQ_kqrBFktqd47yzKwVFDwWIdIEfixBqS90PpZQNWlw$>
:
@@ -71,7 +71,7 @@ jobs:
uses: ***@***.***
with:
distribution: 'corretto'
- java-version: '17'
+ java-version: '21'
What's the reason for changing the Java version used for the regular CI
runs? I don't think the Java version here should have any impact on the new
CI job that you've added.
Also, why is the Java version important? Does something fail if we use Java
17? If so, that should be fixed: Specimin should run under Java 17+.
> I changed the java version to 17. I am encountering a gradle version
deprecated warning post which the CI stops. I am currently fixing the same.
This is what I encountered.
Deprecated Gradle features were used in this build, making it incompatible
with Gradle 9.***.
Thanks and regards
Niharika
…On Wed, Jul 17, 2024 at 12:47 PM Martin Kellogg ***@***.***> wrote:
@NiharikaJamble is this ready to review again? If so, please either post a
comment that says so or use the GitHub UI to "request another review". —
Reply to this email directly, view it on GitHub, or unsubscribe. You are
receiving this
@NiharikaJamble
<https://urldefense.com/v3/__https://github.com/NiharikaJamble__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgZLXdecQ$>
is this ready to review again? If so, please either post a comment that
says so or use the GitHub UI to "request another review".
—
Reply to this email directly, view it on GitHub
<https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2233751945__;Iw!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgpujcy3g$>,
or unsubscribe
<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNHNU2XTBVDBQ475KDDZM2N3RAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTG42TCOJUGU__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQiYZu7LQw$>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Apologies for the incorrect error.Below is the error I wanted to mention
for point 2 in previous mail:
Running ASHE...
64
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:65>Error
executing command: Error: LinkageError occurred while loading main class
edu.njit.jerse.automation.RepositoryAutomationEngine
65
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:66>
java.lang.UnsupportedClassVersionError:
edu/njit/jerse/automation/RepositoryAutomationEngine has been compiled by a
more recent version of the Java Runtime (class file version 65.***), this
version of the Java Runtime only recognizes class file versions up to
61.***
66
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:67>
67
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:68>FAILURE:
Build failed with an exception.
68
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:69>
69
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:70>*
What went wrong:
70
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:71>Execution
failed for task ':runRepositoryAutomation'.
71
<https://github.com/NiharikaJamble/specimin/actions/runs/9969996783/job/27547998599#step:11:72>>
Process 'command
'/opt/hostedtoolcache/Java_Adopt_jdk/17.***.11-9/x64/bin/java'' finished
with non-zero exit value 1
…On Wed, Jul 17, 2024 at 4:11 PM Jamble, Niharika ***@***.***> wrote:
Hi Professor
I am working on two comments that you mentioned. Rest of them are fixed.I
have the following issues/questions regarding the same.
1. Looking through the logs of the CI run, this doesn't appear to be quite
working as intended:
- the resulting accuracy is 0%, which is not what we expect
- the cause might be that the logs indicate that Specimin is failing
to run with some kind of problem related to the auto-formatter Spotless.
Why is that happening?
- the logic uses a deprecated mechanism for storing results set-update.
The logs have instructions for how to use the new, preferred mechanism
("environment files"), which you should look into.
>> the latest working log I had shows 90.07%, Can i get the log for
which you encountered this issue?
2. In .github/workflows/gradle.yml
<https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*discussion_r1679630354__;Iw!!DLa72PTfQgg!LGE3Duu6fOGQFBMz6Fo6YAQPluvpcLEbEIlYqZlBSOvLLgyRQ_kqrBFktqd47yzKwVFDwWIdIEfixBqS90PpZQNWlw$>
:
> @@ -71,7 +71,7 @@ jobs:
uses: ***@***.***
with:
distribution: 'corretto'
- java-version: '17'
+ java-version: '21'
What's the reason for changing the Java version used for the regular CI
runs? I don't think the Java version here should have any impact on the new
CI job that you've added.
Also, why is the Java version important? Does something fail if we use
Java 17? If so, that should be fixed: Specimin should run under Java 17+.
>> I changed the java version to 17. I am encountering a gradle version
deprecated warning post which the CI stops. I am currently fixing the same.
This is what I encountered.
Deprecated Gradle features were used in this build, making it incompatible
with Gradle 9.***.
Thanks and regards
Niharika
On Wed, Jul 17, 2024 at 12:47 PM Martin Kellogg ***@***.***>
wrote:
> @NiharikaJamble is this ready to review again? If so, please either post
> a comment that says so or use the GitHub UI to "request another review". —
> Reply to this email directly, view it on GitHub, or unsubscribe. You are
> receiving this
>
>
> @NiharikaJamble
> <https://urldefense.com/v3/__https://github.com/NiharikaJamble__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgZLXdecQ$>
> is this ready to review again? If so, please either post a comment that
> says so or use the GitHub UI to "request another review".
>
> —
> Reply to this email directly, view it on GitHub
> <https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2233751945__;Iw!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgpujcy3g$>,
> or unsubscribe
> <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNHNU2XTBVDBQ475KDDZM2N3RAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTG42TCOJUGU__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQiYZu7LQw$>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
I see - it looks like ASHE is compiling itself with Java 21, so its binaries don't work with Java 17. That's a good reason.
I'm referring to the log of the run of the CI job on this PR. In particular, I think I was looking at this one: https://github.com/njit-jerse/specimin/actions/runs/9952758388/job/27513834573 |
kelloggm
changed the title
|
@NiharikaJamble CI is not running in this PR because there are conflicts with the |
|
Hi Professor
I did check on that. The error indicates that origin/main is not a branch
present. However that is not the case since we do have a main branch. Also
in the run you mentioned , only the gradle in use for ASHE was of a
different version w.r.t the successful run. Given that gradle versions
also impact the plugins used I have changed the path to clone the
directories into directory of ASHE project so while running the gradle used
is the one mentioned in gradle wrapper of ASHE project.
Also I am unable to run my updated changes in the PR.
So I wanted to request a run before we meet so I could have some outputs
for the meeting
Thanks and regards
Niharika
…On Wed, Jul 24, 2024 at 3:09 PM Martin Kellogg ***@***.***> wrote:
@ NiharikaJamble CI is not running in this PR because there are conflicts
with the main branch that you need to resolve. Once the conflicts are
resolved CI should run again. — Reply to this email directly, view it on
GitHub, or unsubscribe.
@NiharikaJamble
<https://urldefense.com/v3/__https://github.com/NiharikaJamble__;!!DLa72PTfQgg!KcwZzq5c7HrHeRNegCtXRYPOcyflNKQn1MmTtYa-Cs70n7-YA9S880qJbFQJMxpGaVQlrb96gR3Ui15iy28LQZbTQQ$>
CI is not running in this PR because there are conflicts with the main
branch that you need to resolve. Once the conflicts are resolved CI should
run again.
—
Reply to this email directly, view it on GitHub
<https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2248721585__;Iw!!DLa72PTfQgg!KcwZzq5c7HrHeRNegCtXRYPOcyflNKQn1MmTtYa-Cs70n7-YA9S880qJbFQJMxpGaVQlrb96gR3Ui15iy29szH0Erw$>,
or unsubscribe
<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNGYDW2JJJ7SN3A3ES3ZN73WFAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYG4ZDCNJYGU__;!!DLa72PTfQgg!KcwZzq5c7HrHeRNegCtXRYPOcyflNKQn1MmTtYa-Cs70n7-YA9S880qJbFQJMxpGaVQlrb96gR3Ui15iy28fSbHo_w$>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I don't know what you mean by this. There is clearly a conflict in |
|
|
Hi Professor
As per our discussion yesterday I have raised another PR. Can you please
review the same?
Along with this I have noticed by pulling the newly added code the accuracy
has dropped to 88.08 % from 90.07 %. I would like to check my run in the PR
but seems like I cannot do it without your review.( some with write access
needs to approve it)
Thanks & Regards
Niharika Jamble
…On Wed, Jul 24, 2024 at 4:16 PM Martin Kellogg ***@***.***> wrote:
Screen. Shot. 2024-07-24. at. 4. 16. 09. PM. png (view on web) — Reply to
this email directly, view it on GitHub, or unsubscribe. You are receiving
this because you were mentioned. Message ID:
<njit-jerse/specimin/pull/323/c2248822328@ github. com>
Screen.Shot.2024-07-24.at.4.16.09.PM.png (view on web)
<https://urldefense.com/v3/__https://github.com/user-attachments/assets/7efc8508-f537-4e5a-a0e2-c5b1234cc99e__;!!DLa72PTfQgg!IplLMv7iNBC59SAnAqf7C0HVNhy_CVjiEYrvZSVbn6QOUfR9V3rEOAbAOn7KkrhoeIOzHX3pc8DBlw3yxxAYRrqnmQ$>
—
Reply to this email directly, view it on GitHub
<https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2248822328__;Iw!!DLa72PTfQgg!IplLMv7iNBC59SAnAqf7C0HVNhy_CVjiEYrvZSVbn6QOUfR9V3rEOAbAOn7KkrhoeIOzHX3pc8DBlw3yxxCahU--8Q$>,
or unsubscribe
<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNDYOX22LX6M4TCZRSTZOADSJAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYHAZDEMZSHA__;!!DLa72PTfQgg!IplLMv7iNBC59SAnAqf7C0HVNhy_CVjiEYrvZSVbn6QOUfR9V3rEOAbAOn7KkrhoeIOzHX3pc8DBlw3yxxAEAZuhQA$>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
In the future, please request a review directly on the PR that needs to be reviewed, either by tagging me in a comment or by using the "request a review" feature in the GitHub UI.
We've been making changes to Specimin, so it's possible that we've introduced a regression. Please don't worry about the exact value at this time. |
I'm not sure where you're seeing this. Looking at your PR #330, it looks like CI is failing on your added job because there is some missing file: https://github.com/njit-jerse/specimin/actions/runs/10084952318/job/27884789451. You should have sufficient permissions to trigger CI (as seen by the fact that CI does run on #330). I'd prefer not to review a PR until CI passes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change will execute ASHE to check Specimin's accuracy (defined as ability to produce compilable output) on any push or pull requests made to main branch