Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.lang.RuntimeException: Problem using the file: java.nio.file.FileSystemException: path_to_file.scala: Invalid argument #94

Closed
mcenkar opened this issue Mar 3, 2023 · 4 comments
Closed

java.lang.RuntimeException: Problem using the file: java.nio.file.FileSystemException: path_to_file.scala: Invalid argument #94

mcenkar opened this issue Mar 3, 2023 · 4 comments

Comments

@mcenkar
Copy link
Contributor

mcenkar commented Mar 3, 2023 •
edited
Loading

Hi,

when upgrading sbt-avrohugger from 2.2.1 to 2.3.0 I'm faced with following error:

[error] java.lang.RuntimeException: Problem using the file: java.nio.file.FileSystemException: /.../workspace/website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/RegularTradingHourEvent.scala: Invalid argument
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)
[error] 	at avrohugger.format.abstractions.SourceFormat.writeToFile$(SourceFormat.scala:218)
[error] 	at avrohugger.format.SpecificRecord$.writeToFile(SpecificRecord.scala:21)
[error] 	at avrohugger.format.SpecificRecord$.$anonfun$compile$1(SpecificRecord.scala:222)
[error] 	at avrohugger.format.SpecificRecord$.$anonfun$compile$1$adapted(SpecificRecord.scala:222)
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at avrohugger.format.SpecificRecord$.compile(SpecificRecord.scala:222)
[error] 	at avrohugger.generators.FileGenerator$.protocolToFile(FileGenerator.scala:50)
[error] 	at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1(FileGenerator.scala:94)
[error] 	at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1$adapted(FileGenerator.scala:89)
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] (website-server-api / Compile / avroScalaGenerateSpecific) Problem using the file: java.nio.file.FileSystemException: /.../workspace/website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/RegularTradingHourEvent.scala: Invalid argument
[error] Total time: 27 s, completed 3 Mar 2023, 08:22:40

seems like running an extra compile does solve the problem. However it's a bit difficult on CI + it was working fine on 2.2.1.

Did anything change in how sbt-avrohugger needs to be plugged in? My build is probably a bit complicated because it's compiling avdl files extracted from jar by another sbt task, so possible there's some error on my part.

// edit: actually it seems to be a bit nondeterministic as sometimes builds pass just fine
@julianpeeters
Copy link
Owner

Hi @mcenkar, sorry you're experiencing this, and thanks for reporting.

I don't have very helpful insight, unfortunately:

  1. The changes from 2.2.1 to 2.3.0 were very minimal, but possibly the nondeterminism was made visible by a the bump to sbt 1.8.2
  2. sbt-avrohugger has long suffered from a known non-determinism, and, while your symptoms are different, the concurrency workaround may be a long shot.
  3. If your issue is another manifestation of (2), then the current plan is to use a Ref to solve the issue, but that's a little further down the roadmap
  4. I could be way off-base with the above, as I haven't been able to reproduce these issues myself

@mcenkar
Copy link
Contributor Author

mcenkar commented Mar 3, 2023

Actually had a longer look and it's not 2.3.0, I had it even on 2.0.0 - only noticed now, or maybe was just unlucky to get it couple times in a row.

Maybe narrowed down nondeterminism to:

  • sometimes I get this error:
[info] Compiling Avro IDL /.../website-protobuf/server-api/src/main/generated/avro/ListedOptionIdentifiersEvents.avdl to /.../website-protobuf/server-api/target/src_managed/main/compiled_avro
[info] Compiling 2 protobuf files to /.../website-protobuf/server-api/target/src_managed/main
[error] java.lang.RuntimeException: Problem using the file: java.nio.file.NoSuchFileException: /.../website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/ListedOptionIdentifiersEvent.scala
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)

and sometimes it manages to compile 2 files (out of 6 total):

[info] Compiling Avro IDL /.../website-protobuf/server-api/src/main/generated/avro/ListedOptionIdentifiersEvents.avdl to /.../website-protobuf/server-api/target/src_managed/main/compiled_avro
[info] Compiling Avro IDL /.../website-protobuf/server-api/src/main/generated/avro/ListedOptionIdentifiersEventV2.avdl to /.../website-protobuf/server-api/target/src_managed/main/compiled_avro
[info] Compiling 2 protobuf files to /.../website-protobuf/server-api/target/src_managed/main
[error] java.lang.RuntimeException: Problem using the file: java.nio.file.NoSuchFileException: /.../website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/ListedOptionIdentifiersEventV2.scala
[error] 	at scala.sys.package$.error(package.scala:30)

we're pretty much here:

[error] java.lang.RuntimeException: Problem using the file: java.nio.file.NoSuchFileException: /.../website-protobuf/server-api/target/src_managed/main/compiled_avro/com/events/ListedOptionIdentifiersEventV2.scala
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)

https://github.com/julianpeeters/avrohugger/blob/v1.3.0/avrohugger-core/src/main/scala/format/abstractions/SourceFormat.scala#L231

which doesn't quite make much sense to me, maybe to you:

      Files.deleteIfExists(path)
      Files.write(path, contents, StandardOpenOption.CREATE)

first one shouldn't throw an error, second one shouldn't throw NoSuchFileException because it should just create the file.

// edit Actually second would throw this exception if entire folder didn't exist. So it's supposed to write to /a/b/c.txt but there's only /a/ => it will throw an error.

@julianpeeters maybe there's too much deleting somewhere?

full stacktrace for reference:

[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at avrohugger.format.abstractions.SourceFormat.writeToFile(SourceFormat.scala:231)
[error] 	at avrohugger.format.abstractions.SourceFormat.writeToFile$(SourceFormat.scala:218)
[error] 	at avrohugger.format.SpecificRecord$.writeToFile(SpecificRecord.scala:21)
[error] 	at avrohugger.format.SpecificRecord$.$anonfun$compile$1(SpecificRecord.scala:222)
[error] 	at avrohugger.format.SpecificRecord$.$anonfun$compile$1$adapted(SpecificRecord.scala:222)
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at avrohugger.format.SpecificRecord$.compile(SpecificRecord.scala:222)
[error] 	at avrohugger.generators.FileGenerator$.protocolToFile(FileGenerator.scala:50)
[error] 	at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1(FileGenerator.scala:94)
[error] 	at avrohugger.generators.FileGenerator$.$anonfun$fileToFile$1$adapted(FileGenerator.scala:89)
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at avrohugger.generators.FileGenerator$.fileToFile(FileGenerator.scala:89)
[error] 	at avrohugger.Generator.fileToFile(Generator.scala:80)
[error] 	at sbtavrohugger.FileWriter$.$anonfun$generateCaseClasses$8(FileWriter.scala:37)
[error] 	at sbtavrohugger.FileWriter$.$anonfun$generateCaseClasses$8$adapted(FileWriter.scala:34)
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at sbtavrohugger.FileWriter$.generateCaseClasses(FileWriter.scala:34)
[error] 	at sbtavrohugger.SbtAvrohugger$.$anonfun$specificAvroSettings$8(SbtAvrohugger.scala:160)
[error] 	at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:80)
[error] 	at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:153)
[error] 	at sbt.util.Difference.apply(Tracked.scala:414)
[error] 	at sbt.util.Difference.apply(Tracked.scala:394)
[error] 	at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:149)
[error] 	at sbt.util.Difference.apply(Tracked.scala:414)
[error] 	at sbt.util.Difference.apply(Tracked.scala:389)
[error] 	at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:148)
[error] 	at sbtavrohugger.SbtAvrohugger$.$anonfun$specificAvroSettings$7(SbtAvrohugger.scala:161)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] 	at sbt.Execute.work(Execute.scala:291)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1589)

mcenkar added a commit to mcenkar/avrohugger that referenced this issue Mar 3, 2023
@mcenkar
fix: julianpeeters/sbt-avrohugger#94
ffbd5b4 
Files were generated into not-existing folder, added call to create all missing
parent folders.
@mcenkar
Copy link
Contributor Author

mcenkar commented Mar 3, 2023

@julianpeeters maybe there's too much deleting somewhere?

That was actually pretty good guess, if I create all folders on-the-way to actual one I don't get any more errors.

julianpeeters/avrohugger#165

Not sure if there might be another bug - because why it's nondeterministic - can something delete folder when files were already generated - but it does fix problem for me.

@julianpeeters
Copy link
Owner

@mcenkar thanks a ton for looking into this. The fix has been released as avrohugger version 1.3.1 and sbt-avrohugger version 2.3.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@julianpeeters @mcenkar