Mass ccache-related build errors for AT1 PR builds PR builds gnu-8.5.0-openmpi-4.1.6 and cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6 on ascic0193 starting 2025-03-03

[bartlettroscoe]

CC: @trilinos/framework, @sebrowne, @achauphan, @ccober6

Description

As shown in this CDash query showing:

there are mass build errors for the build configurations:

• rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
• rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables

all on the machine ascic0193 starting 2025-03-03.

This impacted many PR build iterations for different PRs #13847, #13849, #13850, #13851, and #13852 so far.

This is showing errors like:

ccache: error: Failed to create directory /fgs/trilinos/ccache/cache/1/5: Permission denied

[bartlettroscoe]

FYI: I pinned this issue to warn other Trilinos developers trying to get their PRs to pass testing.

[bartlettroscoe]

Let's see how many people manually add AT: RETEST before they realize the futility of doing so before this issue is resolved?

[sebrowne]

Thanks for noticing this, I was in meetings most of the day and just saw it. Ticket has been submitted, and I removed 0193 from the pool of test resources. Anything starting from now on is safe.

[sebrowne]

Admins have restored the bad filesystem, and I just re-added ascic0193 to our CI pool. Should be fixed now.