-
Notifications
You must be signed in to change notification settings - Fork 572
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update TriBITS snapshot 2024-01-24 and set Trilinos_SHOW_GIT_COMMIT_PARENTS=ON for PR builds #12695
Update TriBITS snapshot 2024-01-24 and set Trilinos_SHOW_GIT_COMMIT_PARENTS=ON for PR builds #12695
Conversation
Origin repo remote tracking branch: 'github/master' Origin repo remote repo URL: 'github = [email protected]:TriBITSPub/TriBITS.git' Git describe: tribits_start-3375-g9328b340 At commit: commit 2f17d8e766e950378a3d0690ec28973134be7d62 Author: Roscoe A. Bartlett <[email protected]> Date: Wed Jan 24 19:38:16 2024 -0500 Summary: Merge branch 'master' into 597-config-git-version-info
…arent-info (TriBITSPub/TriBITS#597) Main purpose is to pull in the TriBITS PR for getting the git repo parent commit info: * TriBITSPub/TriBITS#598
Wow, 11 hours and the autotester still has not picked up testing of this PR. @achauphan and @sebrowne, I am wondering if it might not be a good idea to turn on Update: I pushed the commit da7de59 which I verified locally turns on this option. |
…riBITSPub/TriBITS#597) This is needed to know the versions of the target and topic branches for PR builds submitted to CDash. This is needed to be able to detect random test failures.
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request. |
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Using Repos:
Pull Request Author: bartlettroscoe |
CC: @achauphan, @sebrowne And we are seeing the parent commit info show for this PR. For example here shows:
while another build here shows:
So we can see that the local merge commit SHA1s are different ( @achauphan and @sebrowne, can one of you please approve this PR so that it can merge? (Crossing fingers there will be no random failures :-)) |
Sweet, it's working on all builds! Interesting that the merge commit SHA1s are different here. That would suggest that time of commit is used for the hashing, which I didn't know. |
That does seem to be the case and that is the behavior we have observed for years with the autotester. NOTE: Once Trilinos adopts GHA, then GH will create the merge commit in the central repo and that same merge commit will be checked out in every GHA runner. |
FYI: Look at the results on CDash here, it appears that one of the seven PR builds never fired off (or is not able to submit to CDash). It is hard to match up the builds (given the huge long build names) but it looks like the missing build is the
All of the other builds have completed and passed everything 🥲 From looking that this query, it appears that that build has not showed up on CDash since 2024-01-24 (and the two most recent builds yesterday did not show any build or test results). @achauphan and @sebrowne, can someone see what happened to that build? (I can't remember where the Jenkins site driving this builds is located.) Update: I created an issue for this with some more data showing how bad this is: |
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
|
Interesting, the last PR testing iteration took 10 hours to come back failed (it started at 12:02 PM and came back failed at 10:03 pm yesterday). I am guessing there is a 10 hours time limit on the Jenkins runners (or the main driver Jenkins job). |
Referencing status update in #12696. Looks like this job was started while all GPU nodes were offline and searched for an available node for about ~7 hours until all nodes GPU nodes were brought back up. Tests on this node were taking ~60s each which ran until the timeout limit. The node that picked up this job was previously turned off due to the same performance issues, but was brought back online along with the other recent offline nodes. I've turned this node back offline. The other nodes seem to be performing as usual so I will add request a RETEST here. |
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Using Repos:
Pull Request Author: bartlettroscoe |
@achauphan, does the autotester block until this last Jenkins job timed out? If so, that would mean that the other active odd-numbers PRs were waiting to be tested while that one Jenkins job was trying to get a job running for 10 hours? I am guessing that the GHA implementation will not have this behavior since it can better schedule jobs across available machines (unlike the current autotester implementation that just uses a simple modulo implementation The Trilinos experience with autotester is a case study in why it is not trivial to write your own CI testing system. |
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
|
FYI, the last PR testing iteration above failed because the But as shown in this query, that |
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Using Repos:
Pull Request Author: bartlettroscoe |
Looks like the |
Heads up, this next AT run is going to fail. Three of the nodes that the following builds were running on went offline in middle of their AutoTester run. |
When it rains, it pours |
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Using Repos:
Pull Request Author: bartlettroscoe |
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
|
Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ achauphan ]! |
Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged |
Merge on Pull Request# 12695: IS A SUCCESS - Pull Request successfully merged |
This TriBITS snapshot pulls in updates to TriBITS 'master' from the TriBITS PRs: