Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProvisionException already exists in repository #978

Open
laeubi opened this issue May 29, 2022 · 9 comments
Open

ProvisionException already exists in repository #978

laeubi opened this issue May 29, 2022 · 9 comments

Comments

@laeubi
Copy link
Member

laeubi commented May 29, 2022

In a parallelized build I see the following exception

Caused by: org.eclipse.equinox.p2.core.ProvisionException: Artifact canonical: osgi.bundle,org.eclipse.core.databinding.property,1.9.0.v20210619-1129 already exists in repository file:<path>/.mvn/
    at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl$AddingArtifactSink.<init> (ArtifactRepositoryBaseImpl.java:402)
    at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.internalNewAddingArtifactSink (ArtifactRepositoryBaseImpl.java:382)
    at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.newAddingArtifactSink (ArtifactRepositoryBaseImpl.java:376)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:325)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.ensureArtifactIsPresentInCanonicalFormat (MirroringArtifactProvider.java:340)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:202)
    at org.eclipse.tycho.repository.local.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:132)
    at org.eclipse.tycho.repository.p2base.artifact.provider.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:87)
    at org.eclipse.tycho.p2.target.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:154)
    at org.eclipse.tycho.p2.resolver.DefaultP2ResolutionResult.lambda$2 (DefaultP2ResolutionResult.java:89)
    at org.eclipse.tycho.p2.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73)
    at org.eclipse.tycho.p2.resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$2 (P2DependencyResolver.java:479)
    at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:66)
    at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:62)
    at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:121)
    at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:62)
    at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:121)
    at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:62)
    at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:142)
    at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:134)
    at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:66)
    at org.eclipse.tycho.p2.resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:499)
    at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:178)
    at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.lambda$resolveProjects$0 (TychoMavenLifecycleParticipant.java:157)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept (ForEachOps.java:183)
    at java.util.stream.SpinedBuffer$1Splitr.forEachRemaining (SpinedBuffer.java:364)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:484)
    at java.util.stream.ForEachOps$ForEachTask.compute (ForEachOps.java:290)
    at java.util.concurrent.CountedCompleter.exec (CountedCompleter.java:746)
    at java.util.concurrent.ForkJoinTask.doExec (ForkJoinTask.java:290)
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec (ForkJoinPool.java:1020)
    at java.util.concurrent.ForkJoinPool.scan (ForkJoinPool.java:1656)
    at java.util.concurrent.ForkJoinPool.runWorker (ForkJoinPool.java:1594)
    at java.util.concurrent.ForkJoinWorkerThread.run (ForkJoinWorkerThread.java:177)

FYI @merks seems similar to what I recently encountered with Oomph

@Bananeweizen
Copy link
Contributor

Sounds a bit like #663, but you probably know better if there is any relation between them.

@laeubi
Copy link
Member Author

laeubi commented May 30, 2022

No this is sadly something different, this error is emitted by P2, while the other is emitted by the OS.

@merks
Copy link
Contributor

merks commented May 30, 2022

Yes, I very occasionally see this problem in Oomph as well. But it's never possible to reproduce because if you try again, it succeeds. I suppose one could manually reproduce it by placing the artifact that will be download at the location where it will end up. Instead of failing though, p2 might handle it more gracefully and produce only a warning/log rather than fail the overall process; but should p2 overwrite the artifact or leave it as is?

@laeubi
Copy link
Member Author

laeubi commented May 30, 2022

p2 might handle it more gracefully and produce only a warning/log rather than fail the overall process; but should p2 overwrite the artifact or leave it as is?

That's the question, I wonder what one would gain here to throw errors, at least at Tycho there is even code to first remove the descriptor and add it afterwards if it already exits, but still there is of course a window where a race could happen.
I think the initial idea was that there should be no intermediate state (e.g. A descriptor is visible to other threads but its contents is overwritten then), tycho uses a "downloadlock" for this but it seems this is not enough here.

@mickaelistria
Copy link
Contributor

should p2 overwrite the artifact or leave it as is?

I believe the fact that there is no clear answer what's best is the reason why an exception was the chosen behavior: at least one can't blame p2 for taking the wrong decision (ie installing an unexpected artifact, or corrupting an existing installation).

Isn't the issue that multiple thread try to download the exact same file at the same time? If so, I imagine we can protect the code against that by putting some MirrorArtifact on hold if they happen to target the same destination as one currently running. Then upon completion of first mirror action, the next MirrorArtifact for this destination would check the target file signature/size and compare it to the ones expexted by p2 and continue if they are a match (which I think is most often the case).

@merks
Copy link
Contributor

merks commented May 30, 2022

No, I don't think multiple threads are trying to download the same artifact. In general, the set of artifact requests is collected and requests are distributed across threads, but that's a comment about how p2 (and Oomph) use a transaction to update a profile. I don't know what Tycho is doing in this regard...

I like the idea of comparing the artifacts. Obviously if they are equal such an issue could be quietly ignored...

@laeubi
Copy link
Member Author

laeubi commented May 30, 2022

The problem is, to compare them one needs to download them, but as the download is rejected because the artifact already exits ...
Anyways I already added support to Tycho to use the sha1 + md5 to decide if an artifact needs to be downloaded again if the properties change.

Also, Tycho here only emulates P2 (the error is from pure Tycho code!) and already has a "download lock" so the problem is probably more that the artifactrepositorymanager making the same thing available to different parts of the code and then it could be happen that things are not in sync anymore.

@richardgroote
Copy link

We're experiencing the same on tycho 3.0.4. Sometimes the build is failing due to the below exception. Running it multiple times again later, works just fine.

Kind regards,,
Richard

Caused by: org.eclipse.equinox.p2.core.ProvisionException: Artifact canonical: osgi.bundle,jcl.over.slf4j,1.7.32 already exists in repository file:/data/buildserver/jenkins/maven/../../maven/product.feature/myfeature/ at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl$AddingArtifactSink.<init> (ArtifactRepositoryBaseImpl.java:402) at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.internalNewAddingArtifactSink (ArtifactRepositoryBaseImpl.java:382) at org.eclipse.tycho.repository.p2base.artifact.repository.ArtifactRepositoryBaseImpl.newAddingArtifactSink (ArtifactRepositoryBaseImpl.java:376) at org.eclipse.tycho.repository.local.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:313) at org.eclipse.tycho.repository.local.MirroringArtifactProvider.ensureArtifactIsPresentInCanonicalFormat (MirroringArtifactProvider.java:333)

@HannesWell
Copy link
Member

I encounter this issue as well with Tycho 4.0.2 frequently and first assumed it is a regression in the new version (but this issue proofs that wrong). We recently also changed to parallel builds so maybe the parallel resolution of dependencies is causing this problem?
The artifact is actually not new and we use a shared .m2 cache (but since that is an artifact from a p2-repo it is probably not cached forever) and AFAICT there was no other job running at the time. So my initial guess is that the parallel resolution is interfering with itself.
Wouldn't it be possible to synchronize the mirror of the same artifact? From the metadata it should be possible to determine if it is the same or not.

 [ERROR] foo.bar: Error while mirroring artifact osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 to the local Maven repositoryArtifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
 java.lang.RuntimeException: foo.bar: Error while mirroring artifact osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 to the local Maven repositoryArtifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
     at io.takari.maven.builder.smart.SmartBuilderImpl.buildProject (SmartBuilderImpl.java:214)
     at io.takari.maven.builder.smart.SmartBuilderImpl$ProjectBuildTask.run (SmartBuilderImpl.java:81)
     at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
     at java.util.concurrent.FutureTask.run (FutureTask.java:264)
     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
     at java.lang.Thread.run (Thread.java:833)
 Caused by: org.eclipse.tycho.p2.repository.MirroringArtifactProvider$MirroringFailedException: Error while mirroring artifact osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 to the local Maven repositoryArtifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:221)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:137)
     at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:89)
     at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:169)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2 (DefaultP2ResolutionResult.java:90)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6 (P2DependencyResolver.java:383)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:69)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:216)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:180)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:82)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:409)
     at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:155)
     at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution (TychoProjectExecutionListener.java:108)
     at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution (CompoundProjectExecutionListener.java:42)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:103)
     at io.takari.maven.builder.smart.SmartBuilderImpl.buildProject (SmartBuilderImpl.java:209)
     at io.takari.maven.builder.smart.SmartBuilderImpl$ProjectBuildTask.run (SmartBuilderImpl.java:81)
     at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
     at java.util.concurrent.FutureTask.run (FutureTask.java:264)
     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
     at java.lang.Thread.run (Thread.java:833)
 Caused by: org.eclipse.equinox.p2.core.ProvisionException: Artifact canonical: osgi.bundle,org.apache.commons.jxpath,1.3.0.v200911051830 already exists in repository file:/home/build/.m2/repository/
     at org.eclipse.tycho.p2.repository.ArtifactRepositoryBaseImpl$AddingArtifactSink.<init> (ArtifactRepositoryBaseImpl.java:412)
     at org.eclipse.tycho.p2.repository.ArtifactRepositoryBaseImpl.internalNewAddingArtifactSink (ArtifactRepositoryBaseImpl.java:392)
     at org.eclipse.tycho.p2.repository.ArtifactRepositoryBaseImpl.newAddingArtifactSink (ArtifactRepositoryBaseImpl.java:386)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.downloadCanonicalArtifact (MirroringArtifactProvider.java:338)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.ensureArtifactIsPresentInCanonicalFormat (MirroringArtifactProvider.java:353)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.makeLocallyAvailable (MirroringArtifactProvider.java:215)
     at org.eclipse.tycho.p2.repository.MirroringArtifactProvider.getArtifactFile (MirroringArtifactProvider.java:137)
     at org.eclipse.tycho.p2.repository.CompositeArtifactProvider.getArtifactFile (CompositeArtifactProvider.java:89)
     at org.eclipse.tycho.p2resolver.TargetPlatformBaseImpl.getLocalArtifactFile (TargetPlatformBaseImpl.java:169)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResult.lambda$addArtifact$2 (DefaultP2ResolutionResult.java:90)
     at org.eclipse.tycho.core.resolver.DefaultP2ResolutionResultEntry.getLocation (DefaultP2ResolutionResultEntry.java:73)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.lambda$newDefaultTargetPlatform$6 (P2DependencyResolver.java:383)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifactFile$2 (ArtifactCollection.java:69)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.osgitools.targetplatform.ArtifactCollection.lambda$addArtifact$3 (ArtifactCollection.java:148)
     at org.eclipse.tycho.core.osgitools.DefaultArtifactDescriptor.getLocation (DefaultArtifactDescriptor.java:72)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.collectExternalDependencies (MavenDependencyInjector.java:216)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.addDependency (MavenDependencyInjector.java:180)
     at org.eclipse.tycho.core.maven.MavenDependencyInjector.injectMavenDependencies (MavenDependencyInjector.java:82)
     at org.eclipse.tycho.p2resolver.P2DependencyResolver.injectDependenciesIntoMavenModel (P2DependencyResolver.java:409)
     at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:155)
     at org.eclipse.tycho.core.maven.TychoProjectExecutionListener.beforeProjectLifecycleExecution (TychoProjectExecutionListener.java:108)
     at org.apache.maven.lifecycle.internal.CompoundProjectExecutionListener.beforeProjectLifecycleExecution (CompoundProjectExecutionListener.java:42)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:103)
     at io.takari.maven.builder.smart.SmartBuilderImpl.buildProject (SmartBuilderImpl.java:209)
     at io.takari.maven.builder.smart.SmartBuilderImpl$ProjectBuildTask.run (SmartBuilderImpl.java:81)
     at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
     at java.util.concurrent.FutureTask.run (FutureTask.java:264)
     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
     at java.lang.Thread.run (Thread.java:833)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants