Unable to resolve target definition when non-IU locations are used #874

ptziegler · 2023-11-03T11:38:05Z

As a follow-up on #867 and based on the example in #675.

Using the nightly build, the following target-platform definition is able to resolve without problems (i.e. the p2 Guava bundle can use the Maven bundle to resolve its dependencies).

<target name="linuxtools" sequenceNumber="0">
	<locations>
		<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
			<repository location="https://download.eclipse.org/linuxtools/update-docker-5.12.0/"/>
			<unit id="com.google.guava" version="0.0.0"/>
		</location>

		<location includeDependencyDepth="none" includeDependencyScopes="compile" includeSource="true" label="DirectFromMaven" missingManifest="error" type="Maven">
			<dependencies>
				<dependency>
					<groupId>com.google.guava</groupId>
					<artifactId>failureaccess</artifactId>
					<version>1.0.1</version>
					<type>jar</type>
				</dependency>
			</dependencies>
		</location>
	</locations>
</target>

The following does not work (with FailureAccess 1.0.2 instead of 1.0.1):

<target name="linuxtools" sequenceNumber="0">
	<locations>
		<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
			<repository location="https://download.eclipse.org/linuxtools/update-docker-5.12.0/"/>
			<unit id="com.google.guava" version="0.0.0"/>
		</location>

		<location includeDependencyDepth="none" includeDependencyScopes="compile" includeSource="true" label="DirectFromMaven" missingManifest="error" type="Maven">
			<dependencies>
				<dependency>
					<groupId>com.google.guava</groupId>
					<artifactId>failureaccess</artifactId>
					<version>1.0.2</version>
					<type>jar</type>
				</dependency>
			</dependencies>
		</location>
	</locations>
</target>

Setting the Guava version doesn't help, either (with Guava 32.1.2.jre and FailureAccess 1.0.2):

<target name="linuxtools" sequenceNumber="0">
	<locations>
		<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
			<repository location="https://download.eclipse.org/linuxtools/update-docker-5.12.0/"/>
			<unit id="com.google.guava" version="32.1.2.jre"/>
		</location>

		<location includeDependencyDepth="none" includeDependencyScopes="compile" includeSource="true" label="DirectFromMaven" missingManifest="error" type="Maven">
			<dependencies>
				<dependency>
					<groupId>com.google.guava</groupId>
					<artifactId>failureaccess</artifactId>
					<version>1.0.2</version>
					<type>jar</type>
				</dependency>
			</dependencies>
		</location>
	</locations>
</target>

What does work, however, is setting the version of FailureAccess to 1.0.1 first (assuming that the Guava version is fixed), resolve the target-platform definition, update the version to 1.0.2 and then resolve it again, all without closing the editor.

I'll try to squeeze those examples into a test case, when I find the time...

The text was updated successfully, but these errors were encountered:

ptziegler · 2023-11-07T08:36:01Z

I had a look at it and my initial approach was to simply add a new test case to LocalTargetDefinitionTests. Except that I'm almost certain that those aren't executed as part of the integration tests...

In addition, those tests would also require a m2e installation, but I'm not sure if it's a good idea to have the dependency from pde to m2e, when m2e also has a dependency to pde.

Any suggestions on what would be a good way to approach this?

laeubi · 2023-11-07T08:38:25Z

Usually a Maven location can be simulated by a Directory location where you place the file in a folder.

Here is an example:

<target name="target_require_other">
   <locations>
      <location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
         <repository location="https://download.eclipse.org/linuxtools/updates-docker-nightly/"/>
         <unit id="com.google.guava" version="0.0.0"/>
      </location>
      <location path="${project_loc:/test.pde}/lib" type="Directory"/>
      </locations>
</target>

Consider the following situation: The SimRel 2023-12 repository contains FasterXML Jackson bundles with version 2.16.0, with the exception of one bundle that is still in 2.15.3. In order to ensure that all those bundles have the same version, the one in 2.15.3 is also contributed via Maven Central (imitated by a local directory), also in 2.16.0. Trying to resolve this target definition fails with: No repository found containing: osgi.bundle, com.fasterxml.jackson.module.jackson-module-jaxb-annotations, 2.16.0 Meaning PDE knows that there is a newer version available, it just doesn't know where to find it... Note that one can work around this problem by explicitly adding the missing bundle to the IU location.

ptziegler · 2023-11-30T20:09:45Z

Usually a Maven location can be simulated by a Directory location where you place the file in a folder.

Here is an example:

<target name="target_require_other">
   <locations>
      <location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
         <repository location="https://download.eclipse.org/linuxtools/updates-docker-nightly/"/>
         <unit id="com.google.guava" version="0.0.0"/>
      </location>
      <location path="${project_loc:/test.pde}/lib" type="Directory"/>
      </locations>
</target>

That did the trick. Thank you a lot!

After trying to create a small reproducer, I ended up with #957. In short, I don't think this problem is exclusive to using an unbound 0.0.0 version, but something that can generally happen when the same bundle is consumed from multiple locations.

ptziegler · 2023-12-01T15:00:41Z

sigh, I think I'm starting to understand why this doesn't work...

Consider the following target-definition:

<target name="target-platform">
	<locations>
		<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
			<repository location="https://download.eclipse.org/releases/2023-12/"/>
			<unit id="org.glassfish.jersey.media.jersey-media-json-jackson" version="2.41.0"/>
		</location>
		<location includeDependencyDepth="none" includeSource="true" missingManifest="generate" type="Maven">
			<dependencies>
				<dependency>
					<groupId>com.fasterxml.jackson.module</groupId>
					<artifactId>jackson-module-jaxb-annotations</artifactId>
					<version>2.16.0</version>
					<type>jar</type>
				</dependency>
			</dependencies>
		</location>
	</locations>
</target>

Which, as already mentioned, fails with the error No repository found containing: osgi.bundle,com.fasterxml.jackson.module.jackson-module-jaxb-annotations,2.16.0

The problem itself is created within the ArtifactRequest class, but more broadly during the planner resolution in line 1104:

eclipse.pde/ui/org.eclipse.pde.core/src/org/eclipse/pde/internal/core/target/P2TargetUtils.java

Lines 1101 to 1107 in 5a3eebe

    
           // execute the provisioning plan 
        
           IPhaseSet phases = createPhaseSet(); 
        
           IEngine engine = getEngine(); 
        
           IStatus result = engine.perform(plan, phases, subMonitor.split(100)); 
        
           if (result.getSeverity() == IStatus.ERROR || result.getSeverity() == IStatus.CANCEL) { 
        
           	throw new CoreException(result); 
        
           }

At this state, the provisioning context looks like this:

Repositories

file:///home/patrick/.p2/pool/
file:/[eclipse-root]/ws/.metadata/.plugins/org.eclipse.pde.core/
file:/[eclipse-root]/runtime-EclipseApplication/.metadata/.plugins/org.eclipse.pde.core/.p2/org.eclipse.equinox.p2.core/cache/
file:/[eclipse-root]/runtime-EclipseApplication/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool
file:/[eclipse-root]/eclipse/.eclipseextension
file:/[eclipse-root]/eclipse/configuration/org.eclipse.osgi/147/data/listener_1925729951/
file:/[eclipse-root]/eclipse/
https://download.eclipse.org/releases/2023-12/

Extra IUs

com.fasterxml.jackson.module.jackson-module-jaxb-annotations 2.16.0

Which already implies why the resolution fails later on. The way the repositories are resolved is by iterating over all target locations and checking whether they implement IUBundleContainer or TargetReferenceBundleContainer. But the MavenTargetLocation (as well as the DirectoryBundleContainer) extends AbstractBundleContainer and is therefore neither of those.

eclipse.pde/ui/org.eclipse.pde.core/src/org/eclipse/pde/internal/core/target/P2TargetUtils.java

Lines 1402 to 1414 in 5a3eebe

    
           for (ITargetLocation container : containers) { 
        
           	if (container instanceof IUBundleContainer) { 
        
           		URI[] repos = ((IUBundleContainer) container).getRepositories(); 
        
           		if (repos == null) { 
        
           			repos = manager.getKnownRepositories(IRepositoryManager.REPOSITORIES_ALL); 
        
           		} 
        
           		result.addAll(Arrays.asList(repos)); 
        
           	} 
        
           	if (container instanceof TargetReferenceBundleContainer targetRefContainer) { 
        
           		ITargetDefinition referencedTargetDefinition = targetRefContainer.getTargetDefinition(); 
        
           		result.addAll(getArtifactRepositories(referencedTargetDefinition)); 
        
           	} 
        
           }

This means that when the IUs from other target locations are added to the provisioning context, they are added without a location where they can be downloaded from.

eclipse.pde/ui/org.eclipse.pde.core/src/org/eclipse/pde/internal/core/target/P2TargetUtils.java

Lines 1081 to 1085 in 3f15a1e

    
           ProvisioningContext context = new ProvisioningContext(getAgent()); 
        
           context.setProperty(ProvisioningContext.FOLLOW_REPOSITORY_REFERENCES, Boolean.toString(true)); 
        
           context.setMetadataRepositories(getMetadataRepositories(target).toArray(URI[]::new)); 
        
           context.setArtifactRepositories(getArtifactRepositories(target).toArray(URI[]::new)); 
        
           context.setExtraInstallableUnits(getAdditionalProvisionIUs(target));

That's also the reason why this problem doesn't show up when one uses a second p2 software site, because then both repositories are considered during the planner resolution.

<target name="target-platform">
	<locations>
		<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
			<repository location="https://download.eclipse.org/releases/2023-09/"/>
			<unit id="org.glassfish.jersey.media.jersey-media-json-jackson" version="2.40.0"/>
			<!-- (implicit) <unit id="com.fasterxml.jackson.module.jackson-module-jaxb-annotations" version="2.15.2"/> -->
		</location>
		<location includeAllPlatforms="false" includeConfigurePhase="true" includeMode="planner" includeSource="true" type="InstallableUnit">
			<repository location="https://download.eclipse.org/releases/2023-12/"/>
			<unit id="com.fasterxml.jackson.module.jackson-module-jaxb-annotations" version="2.15.3"/>
		</location>
	</locations>
</target>

ptziegler · 2023-12-01T15:02:44Z

The solution would therefore be to

(a) Adapt the target locations to extend the classes described above
(b) Generalize the planner resolution to also understand non-repositories.

laeubi · 2023-12-01T15:12:54Z

@ptziegler Option (b) is how it is done for the units (metadata) already...

ptziegler · 2023-12-01T15:28:56Z

@ptziegler Option (b) is how it is done for the units (metadata) already...

Shouldn't then all of this simply work? 🤔

In the end, something clearly doesn't work correctly.

And as a result, this error makes it pretty much impossible to checkout any of our project workspaces, for which we've created tags or release branches over the past few years.
Unless I use an older Eclipse installation, of course...

laeubi · 2023-12-01T15:30:57Z

Shouldn't then all of this simply work? 🤔

It works for the Metadata but not for the Artifact repository as artifact repository is (not yet) implemented.

ptziegler · 2023-12-01T18:55:25Z

It works for the Metadata but not for the Artifact repository as artifact repository is (not yet) implemented.

Where can I see what the metadata looks like?

Regardless, I think the focus right now should be on salvaging the current situation. With the little understanding of the p2 artifact resolution I have, I assume the process works similar to this:

(1) Get the non-IU-locations into the provision context
(2) Define and register Metadata and Artifact repositories for Maven/Directory locations.
(3) Get the extra repositories into the provision context

Regarding (1), I would keep the non-IU locations separate from the IU locations, similar to how it's done with the IUs themselves. It probably makes sense to change the extraIUs from List<IInstallableUnit> to Map<URI, List<IInstallableUnit> or something similar.

Regarding (2) , the factories are contributed via the following extension points:

org.eclipse.equinox.p2.metadata.repository.metadataRepositories
org.eclipse.equinox.p2.artifact.repository.artifactRepositories

The question is whether there's a way to define a proper "suffix" (like the artifact.xml) in order to identify what type of non-IU location we're dealing with.

(3) should be easy enough to do within the provision context...

laeubi · 2023-12-01T19:08:21Z

Extension points wont work as they are global to the framework and not local to the target resolution, also a (static) extension can't know about any possible local target (that even maybe only exits in memory).

Please also note that Maven/Directory locations never supply anything from the P2 world, they simply don't know that as Targetplatform is not P2 based (This is different in Tycho!).

You can look at org.eclipse.pde.internal.core.target.P2TargetUtils.getAdditionalProvisionIUs(ITargetDefinition) to see how the IULocation (what is P2 based) discovers MetaData Units from other target locations and converts them into usable items.

One probably want to create something like Tycho org.eclipse.tycho.core.resolver.target.FileArtifactRepository in memory and add it to the provisioning context and then inject it to the call of org.eclipse.equinox.p2.engine.ProvisioningContext.getArtifactRepositories(IProgressMonitor) to be included.

ptziegler · 2023-12-01T22:37:48Z

One probably want to create something like Tycho org.eclipse.tycho.core.resolver.target.FileArtifactRepository in memory and add it to the provisioning context and then inject it to the call of org.eclipse.equinox.p2.engine.ProvisioningContext.getArtifactRepositories(IProgressMonitor) to be included.

That was also one of the things I had in mind. My apologies for not being clear enough.

…e#874 Given that units from non-IU locations are considered when resolving IUs, it may happen that units are contained in the resolution path, which are not contained by any repository. This may break previously functional target definitions. To resolve this problem, an in-memory artifact and metadata repository is created, based on the metadata of the non-IU locations. Those repositories are then added to the provisioning context for further processing. Amends dd321b5

Given that units from non-IU locations are considered when resolving IUs, it may happen that units are contained in the resolution path, which are not contained by any repository. This may break previously functional target definitions. To resolve this problem, an in-memory artifact and metadata repository is created, based on the metadata of the non-IU locations. Those repositories are then added to the provisioning context for further processing. Amends dd321b5

ptziegler changed the title ~~Unable to resolve PDE dependencies with Maven artifact when 0.0.0 version is used~~ Unable to resolve target definition when non-IU locations are used Dec 1, 2023

laeubi linked a pull request Dec 3, 2023 that will close this issue

Add non-IU artifact repositories to the planner resolution #874 #962

Merged

laeubi closed this as completed in #962 Dec 5, 2023

vhemery mentioned this issue Dec 20, 2023

Target platform resolution fails with certain Maven artifacts eclipse-m2e/m2e-core#1622

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to resolve target definition when non-IU locations are used #874

Unable to resolve target definition when non-IU locations are used #874

ptziegler commented Nov 3, 2023

ptziegler commented Nov 7, 2023

laeubi commented Nov 7, 2023 •

edited

Loading

ptziegler commented Nov 30, 2023

ptziegler commented Dec 1, 2023 •

edited

Loading

ptziegler commented Dec 1, 2023 •

edited

Loading

laeubi commented Dec 1, 2023

ptziegler commented Dec 1, 2023

laeubi commented Dec 1, 2023

ptziegler commented Dec 1, 2023

laeubi commented Dec 1, 2023

ptziegler commented Dec 1, 2023

Unable to resolve target definition when non-IU locations are used #874

Unable to resolve target definition when non-IU locations are used #874

Comments

ptziegler commented Nov 3, 2023

ptziegler commented Nov 7, 2023

laeubi commented Nov 7, 2023 • edited Loading

ptziegler commented Nov 30, 2023

ptziegler commented Dec 1, 2023 • edited Loading

ptziegler commented Dec 1, 2023 • edited Loading

laeubi commented Dec 1, 2023

ptziegler commented Dec 1, 2023

laeubi commented Dec 1, 2023

ptziegler commented Dec 1, 2023

laeubi commented Dec 1, 2023

ptziegler commented Dec 1, 2023

laeubi commented Nov 7, 2023 •

edited

Loading

ptziegler commented Dec 1, 2023 •

edited

Loading

ptziegler commented Dec 1, 2023 •

edited

Loading