Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClosedFileSystemException caused by null in ZipFileSystem.ensureOpen in IntelliJ 2024.2 with Azul JDK for project and Temurin JDK for compile server #1383

Open
fanf opened this issue Aug 13, 2024 · 19 comments

Comments

@fanf
Copy link

fanf commented Aug 13, 2024

steps

  • download IntelliJ 2024.2 community
  • clone rudder repository: git clone https://github.com/Normation/rudder.git
  • cd to rudder/webapp/sources and run ./rudder/rudder-web/src/main/build.sh to init the js/css env so that to avoid latter compilation errors unrelated to the problem
  • create a new project in IntelliJ in intellij from maven pom located in path_of_clone/rudder/webapp/sources/pom.xml
  • download Temurin 21 (https://adoptium.net/), Azul 21 (https://www.azul.com/downloads/#zulu) and JetBrains JDK (https://github.com/JetBrains/JetBrainsRuntime/releases), unarchive, and add them as a JDK in IntelliJ: Project Structure... > Platform Settings > SDK, then + and select the path to the previous root directories of the SDK ; then in "Project Settings > Project", choose either azul or jbr SDK ; Set the language level to 17
  • open IntelliJ settings and :
    • change the incremental compiler to Zinc in Settings... > Build, Exec, Deploy > Compiler > Scala Compiler top of the screen.
    • force the Scala Compile Server to Temurin SDK in the drop down list in Settings... > Build, Exec, Deploy > Compiler > Scala Compiler > Scala Compile Server then JVM > JDK and change stack size to -Xss8m ;
  • try a compilation: CTRL+F9 : it fails with compilation errors.

If you change the incremental compiler to IDEA it works. It used to work with IntelliJ 2024.1.x. It works if Temurin is chosen.

problem

Note: I started by opening a ticket on JetBrains bug tracker (https://youtrack.jetbrains.com/issue/SCL-22939), but the root cause might be in Zinc thus that ticket too.

The compilation fails on the last project (rudder-web) with two stack traces which seems to indicate that the problem might be in Zinc (see at the end).

I found an other project which had a very similar problem: jobrunr/jobrunr#427
The cause was that the classpath resource manager was not thread safe as show in the correction: jobrunr/jobrunr@248e098

Exception 1 :

scala: ## Exception when compiling 140 sources to /home/fanf/java/workspaces/rudder-project/rudder/webapp/sources/rudder/rudder-web/target/classes
java.nio.file.ClosedFileSystemException
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
java.base/java.nio.file.Files.readAttributes(Files.java:1853)
java.base/java.nio.file.Files.size(Files.java:2462)
sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
scala.util.Try$.apply(Try.scala:217)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
com.facebook.nailgun.NGSession.run(NGSession.java:198)

Exception 2 :

scala: Compilation failed when compiling to: /home/fanf/java/workspaces/rudder-project/rudder/webapp/sources/rudder/rudder-web/target/classes
  null
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
  jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
  jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
  jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
  java.base/java.nio.file.Files.readAttributes(Files.java:1853)
  java.base/java.nio.file.Files.size(Files.java:2462)
  sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
  sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
  sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
  sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
  sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
  sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
  sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
  sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
  java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
  sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
  sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
  sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
  scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
  java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
  scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
  sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
  sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
  scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
  scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
  scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
  sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
  scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
  scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
  scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
  sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
  sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
  sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
  sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
  sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
  sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
  sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
  sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
  sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
  sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
  sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
  sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
  sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
  sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
  sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
  sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
  sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
  scala.util.Try$.apply(Try.scala:217)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
  org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
  org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
  org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
  org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
  org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
  org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
  org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
  org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
  org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
  org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
  java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  java.base/java.lang.reflect.Method.invoke(Method.java:580)
  com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
  com.facebook.nailgun.NGSession.run(NGSession.java:198)
@vasilmkd
Copy link
Contributor

Since Maven also uses Zinc through scala-maven-plugin`, it might be possible to reproduce the exception by just using Maven in the terminal with the JetBrains JDK as above.

@fanf
Copy link
Author

fanf commented Aug 13, 2024

And... No.
But maven plugin is extremely linear and non concurrent contrary to intellij that tries to parallelize as much as possible (for our joy), so if it's a threadsafety problem, it might be consistant (but it's just a guessing game here).

@eed3si9n
Copy link
Member

download JetBrains JDK (https://github.com/JetBrains/JetBrainsRuntime/releases)

If this doesn't happen on Zulu or Eclipse Temurin, I don't think we'd work on this issue. If there's some logical error in the concurrency handling, we'd be happy to review pull requests though.

@fanf
Copy link
Author

fanf commented Aug 14, 2024

@eed3si9n Oh !
I didn't test with azul, and actually I have the same error !
So I retested with Temurin... And now I get the error too.
In the meantime, I deleted my whole .idea and started from a fresh one, so perhaps changed with that.
Let me try a with a fresh user, fresh clone, fresh jvm install so that the reproduction step and qualification is actually OK.

@fanf
Copy link
Author

fanf commented Aug 14, 2024

@eed3si9n so, fresh linux user, fresh clone, fresh install of intellij, fresh everything.
I can reproduce the problem with Azul zulu JDK with the updated steps in top post, ie with setting the scala compile server to use temurin 21 and the project to use zulu 21.
If I set the scala compile server to zulu too, then the error disappear. So it seems that the problem arise when the compiler server JDK is not the same as the project.

I can't test setting JBR as a scala compile server JDK because then I get a NoClassDefFoundError: INIT (perhaps a limitation of JBR).

So I don't know if the problem is in IntelliJ or zinc, but I does not seems to be directly linked to JBR.

@fanf fanf changed the title ClosedFileSystemException caused by null in ZipFileSystem.ensureOpen in IntelliJ 2024.2 with jbr_jcef-21.0.3 JDK ClosedFileSystemException caused by null in ZipFileSystem.ensureOpen in IntelliJ 2024.2 with Azul JDK for project and Temurin JDK for compile server Aug 17, 2024
@Friendseeker
Copy link
Member

Friendseeker commented Sep 15, 2024

For any guys investigating the issue but got hit with the following error when running ./rudder/rudder-web/src/main/build.sh

npm error code 1
npm error path /Users/jiahuitan/rudder/webapp/sources/rudder/rudder-web/src/main/node_modules/elm
npm error command failed
npm error command sh -c node install.js
npm error -- ERROR -----------------------------------------------------------------------
npm error
npm error I tried to get `elm` from @elm_binaries/darwin_arm64, but something went wrong.
npm error This can happen if you use the "--omit=optional" or "--no-optional" npm flag, or
npm error if your "node_modules" folder was copied over from a different computer (or VM).

Try first do a npm install elm under the directory rudder/webapp/sources/rudder/rudder-web/src/main/ and that bypasses the issue for me

@Friendseeker
Copy link
Member

Friendseeker commented Sep 15, 2024

@fanf May I ask if the issue is reproducible on Apple Silicon MacBooks?

I am trying to reproduce the issue but I am stuck with the last step "try a compilation: CTRL+F9 : it fails with compilation errors.". When I click Build -> Build Project, nothing happens on my M1 Mac.


Update: I resolved the issue of Build -> Build Project having no response. I just disabled all non essential downloaded IDE plugins.

However on my M1 Mac, "Build Project" succeeded.... I guess a Linux machine is needed to debug the issue then.


Update 2:

I clicked "Build Project" for several times and I reproduced the same stack trace! I shall start my investigation. So apparently the issue is flaky... Further suggesting that the thread safety hypothesis might be true.

scala: ## Exception when compiling 142 sources to /Users/jiahuitan/rudder/webapp/sources/rudder/rudder-web/target/classes
java.nio.file.ClosedFileSystemException
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1635)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:769)
jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:779)
jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:285)
java.base/java.nio.file.Files.readAttributes(Files.java:1853)
java.base/java.nio.file.Files.size(Files.java:2462)
sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
sbt.internal.inc.UncachedStamps.library(Stamp.scala:472)
sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:450)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$2(Stamp.scala:255)
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:251)
sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:451)
sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:425)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.$anonfun$getOrElseUpdate$2(JavaCollectionWrappers.scala:482)
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
scala.collection.convert.JavaCollectionWrappers$JConcurrentMapWrapper.getOrElseUpdate(JavaCollectionWrappers.scala:482)
sbt.internal.inc.InitialStamps.library(Stamp.scala:425)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$19(Incremental.scala:1055)
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:215)
sbt.internal.inc.AnalysisCallback.$anonfun$addProductsAndDeps$1(Incremental.scala:1055)
scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:727)
scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:721)
scala.collection.AbstractIterable.foldLeft(Iterable.scala:935)
sbt.internal.inc.AnalysisCallback.addProductsAndDeps(Incremental.scala:1015)
sbt.internal.inc.AnalysisCallback.getAnalysis(Incremental.scala:949)
sbt.internal.inc.AnalysisCallback.getCycleResultOnce(Incremental.scala:940)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:455)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:265)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:409)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:496)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:396)
sbt.internal.inc.Incremental$.apply(Incremental.scala:170)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.$anonfun$doCompile$3(SbtCompiler.scala:87)
scala.util.Try$.apply(Try.scala:217)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.doCompile(SbtCompiler.scala:85)
org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:17)
org.jetbrains.jps.incremental.scala.local.LocalServer.doCompile(LocalServer.scala:51)
org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:29)
org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:210)
org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:191)
org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:178)
org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:188)
org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:161)
org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:101)
org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
com.facebook.nailgun.NGSession.runImpl(NGSession.java:312)
com.facebook.nailgun.NGSession.run(NGSession.java:198)

@Friendseeker
Copy link
Member

Friendseeker commented Sep 16, 2024

@vasilmkd May I ask if it is possible to access the source code of org.jetbrains.jps.incremental.scala.local.SbtCompiler somewhere?

Here's my interim hypothesis of the issue. SbtCompiler called sbt.internal.inc.IncrementalCompilerImpl.compile in multiple different threads, and in each of these calls, SbtCompiler supplied the same ReadStamps, and SbtCompiler used a non thread-safe implementation of ReadStamps, causing the issue.


Never mind, SbtCompiler actually passed in a thread safe implementation of ReadStamps.... Guess we need to formulate hypothesis 2. Still feel like this is a thread safety issue though.

@fanf
Copy link
Author

fanf commented Sep 16, 2024

@Friendseeker : the keyboard shortcut should be just an alias for "build project", so perhaps the problem somehow depends on some local properties. For reference, even if it likely won't help, I use a Lenovo t14s amd gen 3 with a Ryzen 7 PRO 6850U a'd I use Archlinux.
I will try with the menu "build project" to check it leads to the same issue. Unfortunately I don't have a Mac to test.

@fanf
Copy link
Author

fanf commented Sep 16, 2024

It really feels like a threadsafety issue : the fact that maven (which is seldom parallelized) works, the very similar bug /stack trace linked in the top post that says it was becausethe classpath resource manager wasn't threadsafe, the flakiness, the fact that it used to work with previous intellij version...

Also, it only happens in the "rudder-web" subproject, which has several unique properties :

  • it's the last one in the dependency graph,
  • it's the only "webapp", other are internal libs so it has many more resources files, and a war structure.

@vasilmkd
Copy link
Contributor

Hi 👋🏻

In Settings > Build, Execution, Deployment > Compiler > Scala Compiler > Scala Compile Server you can disable parallelism during compilation.

I assume this will make the exception go away?

@fanf
Copy link
Author

fanf commented Sep 16, 2024

Computers are complicated.

So, I tried what @vasilmkd suggested (unchecked the  compile independant modules in parallel, up to 6 threads), and it did have an effect 🎉

But the effect is that it fails earlier. But not in the first or second module either :
image

The stacktraces look identical.
So it might be something with resources being closed too early.
I don't see what "rudder-template" has that is special, but the order of modules is not what I would have expected (because my human brain would prefer to do "all ldap-inventory sub-tree" before starting one in "rudder" sub-tree). It's a jar, it doesn't even have lots of resources.

@vasilmkd
Copy link
Contributor

I mean, as long as all of the dependencies for rudder-templates have been met, it shouldn't matter what exact algorithm the IDEA build system uses (we don't control this in the Scala plugin, we just get called to compile a module).

Can you please confirm that's the case in the screenshot above?

@fanf
Copy link
Author

fanf commented Sep 16, 2024

Yes totally, the dependency path is OK.
And it looks like it fails when it switch to the test part of the project.

@vasilmkd
Copy link
Contributor

@fanf I messaged you on Discord, let's please discuss some details there. Thank you!

@vasilmkd
Copy link
Contributor

@vasilmkd
Copy link
Contributor

Debugging the issue, the "zip" file in question which throws the ClosedFileSystemException happens to be any .sig file located in the JDK modules (JDK 9+).

This is a really strange issue. I honestly have no clue what we might be doing in IDEA that could be causing this. Any ideas are greatly appreciated.

@vasilmkd
Copy link
Contributor

But these .sig files are only hashed by Zinc when there's a mismatch between the Project JDK and the JDK used to run the Zinc compiler.

Which matches

If I set the scala compile server to zulu too, then the error disappear. So it seems that the problem arise when the compiler server JDK is not the same as the project.

#1383 (comment)

@vasilmkd
Copy link
Contributor

When the JDK running the Zinc compiler differs from the Project JDK, then Zinc is instantiated with an instance of sbt.internal.inc.javac.ForkedJavaCompiler.

unkarjedy pushed a commit to JetBrains/intellij-scala that referenced this issue Nov 21, 2024
…aths #SCL-22939 fixed

- These paths really do not like being accessed through the regular Java IO means, throwing ClosedFileSystemException instances when accessed concurrently, like we do in the Scala compile server.
- sbt/zinc#1383
unkarjedy pushed a commit to JetBrains/intellij-scala that referenced this issue Nov 22, 2024
…aths #SCL-22939 fixed

- These paths really do not like being accessed through the regular Java IO means, throwing ClosedFileSystemException instances when accessed concurrently, like we do in the Scala compile server.
- sbt/zinc#1383


(cherry picked from commit bc0f86d)
unkarjedy pushed a commit to JetBrains/intellij-scala that referenced this issue Nov 26, 2024
…aths #SCL-22939 fixed

- These paths really do not like being accessed through the regular Java IO means, throwing ClosedFileSystemException instances when accessed concurrently, like we do in the Scala compile server.
- sbt/zinc#1383
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants