Failing Albany builds due to changes in Trilinos having to do with the communicators

[ikalash]

The nightly spack build is failing due to the following error:

420    /tmp/ikalash/spack-stage/spack-stage-albany-develop-jfp3ybpbdshnwqb
            uck3xssyfjmdmjwmq/spack-src/src/disc/stk/Albany_STKDiscretization.c
            pp: In member function 'void Albany::STKDiscretization::setupExodus
            Output()':
  >> 421    /tmp/ikalash/spack-stage/spack-stage-albany-develop-jfp3ybpbdshnwqb
            uck3xssyfjmdmjwmq/spack-src/src/disc/stk/Albany_STKDiscretization.c
            pp:2311:63: error: invalid conversion from 'Albany_MPI_Comm' {aka '
            int'} to 'stk::ParallelMachine' {aka 'ompi_communicator_t*'} [-fper
            missive]
     422     2311 |         new stk::io::StkMeshIoBroker(getMpiCommFromTeuchosC
            omm(comm)));
     423          |                                      ~~~~~~~~~~~~~~~~~~~~~~
            ~~~^~~~~~

I suspect it's from the PR merge that was done on 12/22 by @bartgol . Could you please have a look @bartgol (or anyone else)?

It's likely the issue will show up in other nightlies. They just happened to all have failed last night due to some Trilinos failures. I'm fixing it now, so we should know tomorrow.

[ikalash]

I just confirmed that this shows up in a regular Albany build w/o spack: https://sems-cdash-son.sandia.gov/cdash/viewBuildError.php?buildid=43483 .

[ikalash]

So, it actually looks like this is due to Trilinos, not Albany. I attempted a fix and am not getting undefined reference errors to utilities in Albany_CommUtils.hpp: https://sems-cdash-son.sandia.gov/cdash/viewBuildError.php?buildid=43484 . I verified that if I check out master Trilinos instead of develop, the problem goes away. After the break, it would be great if someone (@mperego ?, @bartgol ?, @jewatkins ?) can look into this and get the issue resolved. I sadly do not have the bandwidth right now. I will probably switch the nightlies to use master Trilinos for the time being so that they are not failing.

As an aside, I don't know whose bright idea it was to merge in a bunch of changes to Trilinos that literally broke everything on multiple levels the day before the shutdown...

[ikalash]

Probably my fix 97930f4 can/should be removed after understanding why there are problems now with the Albany_CommUtilities.cpp and coming up with a real fix.

[jewatkins]

Just a wild guess but maybe the tribits snapshot is clashing with our cmake config. Last passing commit ids I see is Albany: 9c32d47, Trilinos: 0270d8bd4cd so it looks like it was after some of the epetra depr. went in.

[jewatkins]

Agree, not the best thing to happen right before the shutdown... I think I recall a similar situation happened a few years ago...

[ikalash]

@jewatkins : I agree that the tribits clashing is likely the culprit, esp. given that they just pushed a bunch of changes related to tribits. I feel like we should triage it a bit on the Albany side before we open an issue. If you'd like to volunteer for this (after the break), please speak up.

[jewatkins]

I haven't really been following the changes that have been going into Trilinos/Albany but I can try a build after the break if no one else volunteers and the builds are still broken.

[bartlettroscoe]

I agree that the tribits clashing is likely the culprit,

@ikalash, what do you mean by "TriBITS crashing"? Do you mean the HDF5 disable bringing down the configure trilinos/Trilinos#11426?

[ikalash]

I'm not sure what exactly is going on, quite honestly. We made no changes to Albany but are getting undefined reference errors (https://sems-cdash-son.sandia.gov/cdash/viewBuildError.php?buildid=43487) if we build against Trilinos develop now. It doesn't happen when I build against Trilinos master. @lxmota is finding that when he builds LCM against a Trilinos build with MPI enabled, Albany says there was no MPI in Trilinos. Perhaps you have ideas on how your changes could lead to this behavior? I think realistically we will not get the chance to troubleshoot this until after the shutdown on the Albany side.

[bartlettroscoe]

I think realistically we will not get the chance to troubleshoot this until after the shutdown on the Albany side.

@ikalash, okay, let me just put in a PR to back out the Trilinos PR trilinos/Trilinos#11380 for now. I will just need to get someone to approve it.

[bartlettroscoe]

@ikalash, I just created trilinos/Trilinos#11428 to revert the changes. I just need a Trilinos GitHub project member to approve the PR and it will merge after PR testing completes.

[ikalash]

Thanks! I'm happy to approve it, if you add me to the list of possible approvers. It doesn't seem I can approve it when I click on it right now.