Update integrated test baseline storage #3044
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #3044 +/- ##
========================================
Coverage 53.72% 53.72%
========================================
Files 994 994
Lines 84687 84687
========================================
Hits 45497 45497
Misses 39190 39190 ☔ View full report in Codecov by Sentry. |
cssherman
added
type: testing
cssherman
force-pushed
the
feature/sherman/baselineStorage
branch
from
c43b760 to
e5b6cb1
Compare
cssherman
added
the
ci: run integrated tests
cssherman
force-pushed
the
feature/sherman/baselineStorage
branch
2 times, most recently
from
1639947 to
14fcf54
Compare
| # Rebaseline and pack into an archive | ||
| # Note: Keep a copy of the run logs, so that the correct version will be packed | ||
| mv integratedTests/TestResults integratedTests/TestResults_backup | ||
| integratedTests/geos_ats.sh -a rebaselinefailed | ||
| mv integratedTests/TestResults integratedTests/TestResults_rebaseline | ||
| mv integratedTests/TestResults_backup integratedTests/TestResults | ||
| integratedTests/geos_ats.sh -a pack_baselines --baselineArchiveName ${DATA_EXCHANGE_DIR}/baseline_${DATA_BASENAME_WE}.tar.gz --baselineCacheDirectory ${DATA_EXCHANGE_DIR} | ||
|
|
There was a problem hiding this comment.
maybe I missed something but are we always doing this on the CI? Or how are we triggering the baselines update once we verify the diffs are legitimate?
There was a problem hiding this comment.
actually is this just packing them but not pushing them?
There was a problem hiding this comment.
I think that the pattern is to pack and upload references every time we run the integrated tests (storage does not cost too much and we have a cleaning policy, so it's more than all right.
Once the package is created, then it's pushed at another step of the process, directly from within a GHA step.
That prevents us from installing the GCP toolchain in our docker envs.
Also the same upload step is used for both integrated tests and geos bundles indistinctly.
There was a problem hiding this comment.
Okay, makes sense. But do we really need pack and push after every commit? I am not so much worried about space, more about keeping track about what s the latest valid version of the baselines. Like I have missed how we are keeping track of that.
There was a problem hiding this comment.
We only push when it's computed, so everybody's able to check the results.
I'd say that the latest valid version of the baseline will be defined in a yaml file (or equivalent), like today it's a git commit hash (to the integratedTests repo).
There was a problem hiding this comment.
yeah, well, people kind of set the label and then leave it so for some PR we will basically have a push for each commit but it's probably not that bad .
There was a problem hiding this comment.
Yes, this will probably happen (every time the run integrated tests label is there, at least).
I fine with it.
|
|
||
|
|
||
| .. note:: | ||
| Baseline files are stored using the git LFS (large file storage) plugin. | ||
| If you followed the suggested commands in the quickstart guide, then your environment is likely already setup. | ||
| Integrated tests within GEOS CI pipeline are run on a shared machine, and may take up to 30 minutes to complete. It may take some time for the tests to begin if the machine is in use by other developers. |
| To add a new set of tests, create a new folder under the `GEOS/inputFiles` directory. | ||
| This folder needs to include at least one *.ats* file to be included in the integrated tests. | ||
| Using the sedov example, after creating *sedov.ats* the directory should look like |
There was a problem hiding this comment.
at some point it would be nice to enforce the smoke / benchmark nomenclature but I guess that we are not fully there yet.
…/GEOS into feature/sherman/baselineStorage
CusiniM
added
ci: run CUDA builds
…/GEOS into feature/sherman/baselineStorage
src/docs/sphinx/developerGuide/Contributing/IntegratedTests.rst
Outdated
Show resolved
Hide resolved
src/docs/sphinx/developerGuide/Contributing/IntegratedTests.rst
Outdated
Show resolved
Hide resolved
geosPythonPackages PR: GEOS-DEV/geosPythonPackages#14