HSEARCH-3319 WIP: Type-safe field references v2

[marko-bekhta]

https://hibernate.atlassian.net/browse/HSEARCH-3319

[marko-bekhta]

Even though I've added these withRoot(..) methods to factories, I am not sure that they make much sense with the approach references are taking...

Assuming we somehow address the paths, the factory would still point to the same <SR> root type, as the properties in the embedded objects of the "generated" metamodel will use them:

public static class EntityA_ {
	public static ValueFieldReference1<EntityA_, String, String, String> stringA;

	public static SomeNestedObject_ someNestedObject;

	public static RootReferenceScope<EntityA_, EntityA> scope;

	static {
		stringA = ValueFieldReference1.of( "stringA", EntityA_.class, String.class, String.class, String.class );
		someNestedObject = new SomeNestedObject_();

		scope = RootReferenceScopeImpl.of( EntityA_.class, EntityA.class );
	}

	private static class SomeNestedObject_ implements ObjectFieldReference<EntityA_> {

		public final ValueFieldReference1<EntityA_, String, String, String> nestedStringA;

		public SomeNestedObject_() {
			this.nestedStringA = ValueFieldReference1.of( "someNestedObject.stringA", EntityA_.class, String.class, String.class, String.class );
		}

		@Override
		public String absolutePath() {
			return "someNestedObject";
		}

		@Override
		public Class<EntityA_> scopeRootType() {
			return EntityA_.class;
		}
	}
}

If we go with the different <SR> for nestedStringA then we wouldn't be able to use it in a regular case e.g.:

...
.where( f -> f.match().field( EntityA_.someNestedObject.nestedStringA ).matching( "a" ) )

Also, if we change the SR on withRoot call, the current idea for these embedded objects to create a class-per-field would not fly 😔.

If we ignore the <SR> type change, then any property can be passed to the factory methods, which makes the static metamodel pointless 😔 (in this case)

[yrodiere]

Ideally, I'd expect after using withRoot(), people using the factory would use references such as SomeNestedObject_.nestedStringA, with nestedStringA being a static field. Just like they would on root types.

So you'd have two classes per object field:

• SomeNestedObject$Absolute (the name doesn't matter much, but must not conflict with other classes), which would be the type of EntityA_.someNestedObject, and which would expose non-static "sub" fields of type ValueFieldReference<EntityA_, ...>.
• SomeNestedObject_ which would be the type users of withRoot() would rely on, and which would expose non-static "sub" fields of type ValueFieldReference<SomeNestedObject_, ...>.

I don't think you need an inheritance relationship between these two classes, but you may need to redefine ObjectFieldReference to have two generic type parameters: the first being its "holding" type, and the second being the SR of factories obtained by calling withRoot.

That being said... Since SomeNestedObject_ is completely specific to the nestedString field of EntityA, I'm wondering how useful withRoot() would be. The main point of withRoot is to obtain a factory that can be passed to some util method, which is then able to build a predicate. That method is supposed to be used with different object fields that are assumed to have a similar schema.
If that method works for a specific object field only (with a specific metamodel type, e.g. SomeNestedObject_), then... what's the point?

I suppose it all comes down to the "mapped superclass"/inheritance support I mentioned the other time. If we could make sure that object field metamodel uses inheritance, i.e. the metamodel type EntityB_.SomeNestedObject_ extends EntityA_.SomeNestedObject_ as soon as EntityB extends EntityA, then you'd at least be able to create a root from EntityB_.someNestedObject and then use EntityA_.SomeNestedObject_.nestedStringA on the resulting factory.

Union types at the object field level could be a thing too, but that seems... hard. We'd essentially need SomeNestedObject_ to extend all its union types. That means multiple inheritance, which means SomeNestedObject_ must be an interface. Possible, since interfaces can have static fields, but needs investigation.

[marko-bekhta]

Hmm ok I've come up with the next example. Assuming we have a model as

@Indexed
	public static class EntityA {

		@IndexedEmbedded(includePaths = { "a", "b" })
		public EmbeddedThing embA;

		@IndexedEmbedded(includePaths = { "b", "c" })
		public EmbeddedThing embB;
	}

	public static class EmbeddedThing {
		@FullTextField
		String a;
		@FullTextField
		String b;
		@FullTextField
		String c;
	}

and want to have an util to filter by smth.b using this withRoot approach...

we can generate the metamodel as:

	public static class EntityA_ {

		public static a_ embA;
		public static b_ embB;

		// These two are for the absolute paths: 
		public static class a_ extends ObjectFieldReference<EntityA_, EntityA_a_>{
			public static ValueFieldReference<EntityA_, String, String, String> a; // path = embA.a
			public static ValueFieldReference<EntityA_, String, String, String> b; // path = embA.b
		}

		public static class b_ extends ObjectFieldReference<EntityA_, EntityA_b_>{
			public static ValueFieldReference<EntityA_, String, String, String> b; // path = embB.b
			public static ValueFieldReference<EntityA_, String, String, String> c; // path = embB.c
		}
	}

	// These two are for "withRoot"/  relative paths 
	public static class EntityA_a_ {
		public static ValueFieldReference<EntityA_a_, String, String, String> a; // path = a
		public static ValueFieldReference<EntityA_a_, String, String, String> b; // path = b
	}

	public static class EntityA_b_ {
		public static ValueFieldReference<EntityA_b_, String, String, String> b; // path = b
		public static ValueFieldReference<EntityA_b_, String, String, String> c; // path = c
	}

That won't help much for the with withRoot ... so we can try:

	// ideally I guess we'd want to generate:
	public static class EntityA_a_b_common_ {
		public static ValueFieldReference<EntityA_a_b_common_, String, String, String> b; // path = b
	}

	public static class EntityA_a_ extends EntityA_a_b_common_ {
		public static ValueFieldReference<EntityA_a_, String, String, String> b; // path = b
	}
	
	public static class EntityA_b_ extends EntityA_a_b_common_ {
		public static ValueFieldReference<EntityA_b_, String, String, String> c; // path = c
	}
	
	// and then the util method could use  EntityA_a_b_common_ with access to `b` 
	//   and we pass withRoot(EntityA_.embA) withRoot(EntityA_.embB) to the util method ... 

that could probably work.. but if I understand it correctly... currently that withRoot is not tied to a particular embedded type, meaning we can have:

public static class EntityA {
	@IndexedEmbedded
	EmbeddedThing1 e1;
	@IndexedEmbedded
	EmbeddedThing1 e2;
}

public static class EmbeddedThing1 {
	@FullTextField
	String a;
	// some other fields
}

public static class EmbeddedThing2 {
	@FullTextField
	String a;
	// some other fields maybe different from EmbeddedThing1
}

public static class EntityB {
	@IndexedEmbedded
	EmbeddedThing3 e3;
}

public static class EmbeddedThing3 {
	@FullTextField
	String a;
	// some other fields maybe different from EmbeddedThing1/EmbeddedThing2
}

//util method
private SearchPredicate matchFirstAndLastName(SearchPredicateFactory f, String avalue) {
	return f.match().field( "a" ).matching( avalue ).toPredicate();
}

it is ok now to pass any of

withRoot("e1");
withRoot("e2");
withRoot("e3");

but I'm not sure how we'd tie them all to something in the metamodel to be used in that util... 😕

[yrodiere]

but I'm not sure how we'd tie them all to something in the metamodel to be used in that util... 😕

Essentially, to go beyond simple entity inheritance, we'll need to introduce union types at the object field level.

That's what I was talking about here:

Union types at the object field level could be a thing too, but that seems... hard. We'd essentially need SomeNestedObject_ to extend all its union types. That means multiple inheritance, which means SomeNestedObject_ must be an interface. Possible, since interfaces can have static fields, but needs investigation.

[marko-bekhta]

ok next iteration 🙈 I've tried to look at the interfaces and whether we can somehow use them to our advantage. This time, the model is:

@MappedSuperclass
public static class MappedSuperclassThing {
	@Id
	Long id;

	@FullTextField
	String a;
}

@Indexed
@Entity
public static class ContainingA extends MappedSuperclassThing {
	@IndexedEmbedded
	EmbeddedThing1 e1;
	@IndexedEmbedded
	EmbeddedThing1 e2;
}

public static class EmbeddedThing1 {
	@FullTextField
	String a;
	@FullTextField
	String b;
	// some other fields
}

public static class EmbeddedThing2 {
	@FullTextField
	String a;
	// some other fields maybe different from EmbeddedThing1
}

@Entity
@Indexed
public static class ContainingB extends MappedSuperclassThing{
	@IndexedEmbedded
	EmbeddedThing3 e3;
}

public static class EmbeddedThing3 {
	@GenericField
	Integer a;
	// some other fields maybe different from EmbeddedThing1/EmbeddedThing2
}

There are string/int fields a and some embeddables. Ultimately, we'd want to be able to have a util method as:

public static SearchPredicate utilMethodForPredicate(SearchPredicateFactory<? extends Property_String_a_> factory) {
	return factory.match().field( Property_String_a_.a ).matching( "a" ).toPredicate();
}

where we want to accept the factories for "scope roots" that have a string property a in them. Which can be based on ContainingA/ContainingB/EmbeddedThing1/EmbeddedThing2. For the util to work a should be a part of the Property_String_a_:

interface Property_String_a_ {
	ValueFieldReference1<Property_String_a_, String, String, String> a = ....;
}

then we can have the metamodel as:

public static class MappedSuperclassThing_ implements Property_String_a_ {

	public static HibernateOrmRootReferenceScope<MappedSuperclassThing_, MappedSuperclassThing> scope;

	static {
		//a = ValueFieldReference1.of( "a", MappedSuperclassThing_.class, String.class, String.class, String.class );

		scope = RootReferenceScopeImpl.of( MappedSuperclassThing_.class, MappedSuperclassThing.class );
	}
}

public static class ContainingA_ extends MappedSuperclassThing_ {
	public static e1_.Absolute e1;
	public static e2_.Absolute e2;

	public static HibernateOrmRootReferenceScope<ContainingA_, ContainingA> scope;

	static {
		e1 = new e1_.Absolute();
		e2 = new e2_.Absolute();

		scope = RootReferenceScopeImpl.of( ContainingA_.class, ContainingA.class );
	}

	public static class e1_ implements ObjectFieldReference<ContainingA_>, Property_String_a_, Property_String_b_ {

		@Override
		public String absolutePath() {
			return "e1";
		}

		@Override
		public Class<ContainingA_> scopeRootType() {
			return ContainingA_.class;
		}

		static class Absolute extends e1_ {
			public static ValueFieldReference1<ContainingA_, String, String, String> a;
			public static ValueFieldReference1<ContainingA_, String, String, String> b;

			static {
				a = ValueFieldReference1.of( "e1.a", ContainingA_.class, String.class, String.class, String.class );
				b = ValueFieldReference1.of( "e1.b", ContainingA_.class, String.class, String.class, String.class );
			}
		}
	}

	public static class e2_ implements ObjectFieldReference<ContainingA_>, Property_String_a_, Property_String_b_ {

		@Override
		public String absolutePath() {
			return "e2";
		}

		@Override
		public Class<ContainingA_> scopeRootType() {
			return ContainingA_.class;
		}

		static class Absolute extends e2_ {
			public static ValueFieldReference1<ContainingA_, String, String, String> a;
			public static ValueFieldReference1<ContainingA_, String, String, String> b;

			static {
				a = ValueFieldReference1.of( "e2.a", ContainingA_.class, String.class, String.class, String.class );
				b = ValueFieldReference1.of( "e2.b", ContainingA_.class, String.class, String.class, String.class );
			}
		}
	}
}

Since we use ? super SR we can do

SearchScope<ContainingA_, ContainingA> scope = ContainingA_.scope.create( searchSession );

searchSession.search( scope )
		.select( f -> f.field( ContainingA_.a ) )
		.where( f -> utilMethodForPredicate( f ) )
		.fetchHits( 20 )

as ContainingA_ extends Property_String_a_ it works both for projection and util.
Hence, I'm thinking - we can create an interface-per-filed, where we reuse the interfaces with the same name and capabilities(traits)...

[marko-bekhta]

Obviously, JQAssistant does not like these changes 😔 😃. However, I couldn't come up with an alternative to allow scope to be created from the "generated" static metamodel classes. Unless we generate a model specific to the mapper 😖 😕

@yrodiere if you can take a quick look at the approach in this PR whenever you have some free time 😃

[yrodiere]

Unless we generate a model specific to the mapper 😖 😕

That seems like a very valid solution to me?

Not for every Hibernate Search type in the metamodel, of course -- i.e. I wouldn't make ValueFieldReference specific to each mapper. But it seems perfectly fine to me to give EntityA_.scope a mapper-specific type.

[marko-bekhta]

But it seems perfectly fine to me to give EntityA_.scope a mapper-specific type.

mm ok, we'd need to tell the plugin generating these to know which mappers are expected and then have

EntityA_.ormScope
EntityA_.standaloneScope

we probably can detect that from the dependencies (which mappers are present) and have an explicit property if needed I suppose 🤔

[yrodiere]

mm ok, we'd need to tell the plugin generating these to know which mappers are expected and then have

The plugin would already need to know, since it needs to start Hibernate Search in offline mode to create backends and their index descriptors.

[yrodiere]

(Because, yeah, annotation parsing is not enough: there are binders to take into account, and programmatic mapping :x)

[sonarqubecloud]

Quality Gate failed

Failed conditions
78.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

[marko-bekhta]

Hey @yrodiere 😃 the approach we discussed last Friday seems to work well so far. (see the new inline comments or just take a look at the FieldReferenceIT )

One thing I'm not sure how to deal with is named predicates. They are defined in binders, where there's no notion of search scope. Since binders would define the index structure, does it make sense to use the generated metamodel in them? as its generation depends on the binder ... 🤔 🙈

[marko-bekhta]

The entityName-underscore pattern for these "generated" classes won't work because of the ORM, even if that model is not generated... sooo should we prepend the underscore, use some prefix/suffix oooor do something else?

[yrodiere]

I'd suggest looking at what's been done for Jakarta Data, because we need to avoid conflict with that as well.

Maybe SearchContainingA? ContainingA_Search?

[marko-bekhta]

I thought about the naming for this constant, and INDEX seemed like an ok alternative to INSTANCE, and as the comment suggests, we can make this configurable and let the user pick the name.

[yrodiere]

But then for union scopes, you'd have... INDEXES?

Wouldn't SCOPE be a better alternative? Then you'd have ContainingA__ implement SearchScopeReferenceWhatever.

[yrodiere]

Or SCOPE_REF, or a static scopeRef(), or...

[marko-bekhta]

These are examples of intersection interfaces. I'm thinking that it makes sense to keep the trait intersection interface as an inner class, then if there are multiple intersection interfaces per one scope (ContainingA_) there's no ambiguity.

[yrodiere]

Are you sure there's a need to have special handling for the TraitsInteresection interface?

I mean, we already need to generate interfaces/classes for combination of traits, so we already assume something will be generating a FieldProjectionAndMatchPredicateAndExistsPredicateAndQueryStringPredicateFieldReference; can't we use that here, too?

Unrelated note: naming these "trait combination" interfaces will be a challenge in itself.

[marko-bekhta]

Are you sure there's a need to have special handling for the TraitsInteresection interface?

thinking more about it, yeah we can probably reuse the interfaces... I was thinking about a scenario:

@KeywordField(projectable = Projectable.YES, sortable = Sortable.NO)
private String title;

@KeywordField(projectable = Projectable.NO, sortable = Sortable.YES)
private String title;

where the intersection would be just predicates and we wouldn't have it since it is not needed for an actual field, but we could just keep creating these combinations and add them to a lookup for reuse, so +1 😃

Unrelated note: naming these "trait combination" interfaces will be a challenge in itself.

yeah... At some point I was thinking to have a sorted list of trait interfaces and then "map" them to some numbered keys, e.g. T1, T2... Tn, and the name of a combination will be based on these keys, e.g. T1T5T24FieldReferece

[marko-bekhta]

So, in the new commit, I've tried adding a Maven plugin that would generate things. The approach I've taken here is to have model+ search config classes in their own jar that is passed as a dependency to the Maven plugin. The plugin then starts the Search. I've used some of our test helpers to start things since, well, it's just to test how it'll work.

It generates something like:

package org.hibernate.search.metamodel.generator.model;

class MyEntity__ {

	public String text;	
}

package org.hibernate.search.metamodel.generator.model;

class MyProgrammaticEntity__ {

	public String number;
	public String text;	
}

And since it happens on the generate sources phase, these are getting compiled automatically by Maven itself.

As for the plugin configuration:

<configuration>
    <annotatedTypes>
        <type>org.hibernate.search.metamodel.generator.model.MyEntity</type>
        <type>org.hibernate.search.metamodel.generator.model.MyProgrammaticEntity</type>
    </annotatedTypes>
    <properties>
        <hibernate.search.mapping.configurer>org.hibernate.search.metamodel.generator.model.MyConfigurer</hibernate.search.mapping.configurer>
    </properties>
</configuration>

There's a way to pass any properties that should be "forwarded" to the Search through properties. In this case, I've tried passing a configurer to apply some programmatic mapping 🙈.

now.... next thing I'd want to try is to run the plugin in the same module that the model is. From what I've seen so far, it is easy to get the source root, and the dependencies (things that were missing in the AP approach).

[hibernate-github-bot]

Thanks for your pull request!

This pull request does not follow the contribution rules. Could you have a look?

❌ The PR title or body should list the keys of all JIRA issues mentioned in the commits
    ↳ Issue keys mentioned in commits but missing from the PR title or body: [HSEARCH-5300]
❌ The pull request description must contain the license agreement text.
    ↳ The description of this pull request must contain the following license agreement text:

----------------------
By submitting this pull request, I confirm that my contribution is made under the terms of the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0.txt)
and can be relicensed under the terms of the [LGPL v2.1 license](https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt) in the future at the maintainers' discretion.
For more information on licensing, please check [here](https://github.com/hibernate/hibernate-search/blob/main/CONTRIBUTING.md#legal).

----------------------

› This message was automatically generated.

[sonarqubecloud]

Quality Gate failed

Failed conditions
72.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[marko-bekhta]

Hey @yrodiere,
I've added the AP experiments to this branch as in the end we'd want to use the interfaces introduced here to generate that model.... as if the PR wasn't big enough already 😄, but the experiments are all in a single commit, and I've added comments to the places I think are of interest. There's a lot of boring code similar to the one we have in the annotation processors, as these just work with a different model.

the next steps, as I see it, are:

• keep adding the missing bits (e.g. type level binders etc.). As I encounter some new code example from the docs, or just when I remember some missing part, I add it to the test and then see what has to be added to the processor to get it working.
• write a simple class from the model.
• take care of "various backends". I think that we'd just add all backends (lucene,elasticsearch) and have a config property for the annotation processor to let the user specify the backend to use (or even both).
• deal with the ORM mapper. I don't think that it would be feasible to make ORM somehow generate the model from javax.lang.model.element types. Hence, my thinking here is to add a specific processor that reads the ORM's @Id and treats them as @DocumentId; ignore the access type for now 🙈

[marko-bekhta]

The main idea behind this approach is that we are starting a standalone mapper with all possible "annotation discovery" disabled since an annotation processor won't play nice with here 😃.
Then there is a mapping configurer that iterates through the @Indexed entities and uses programmatic mapping to build the model.

[marko-bekhta]

for now I just print the fields out so it's easy to see what is produced, but I've already started looking into how to generate classes. From what I see, Hibernate ORM/Tools are writing the classes with custom code, I also found this https://github.com/palantir/javapoet and was considering whether it would make sense to pull a dependency, or just do yet another custom class writer 😄

[marko-bekhta]

had to "reinvent" the wheel 😄 as these repeatable annotations are wrapped in the list one, even if the user didn't do that explicitly 🫨 (just adapted the code used in the mapper to expend these repeatable annotations)

[marko-bekhta]

you'll see a few options to get the types by this introspector...
to leverage all built-in bridges etc, we have to pass them the Java types they use in a regular run, rather than typelements. There's a list of "loadable" types (I'm planning to add a test to make sure that it is kept in sync with what is actually present). When we are going through the model and encounter a property of this loadable type, we will end up loading a class and creating a regular type model.
for other cases, when we are working with the type elements, there's ProcessorPojoRawTypeModel and ProcessorPojoPropertyModel. These types implement most of the API, except the annotations, as those aren't available and we are working with the annotation mirrors...

[marko-bekhta]

In general all these processors are the same as the ones in the mapper, but they are based on the javax.lang.model.* types. The other difference is that they have two process methods, one for the regular use of the search annotations and the other one for "injectable fields" in the binders.

[marko-bekhta]

at first I thought we could just continue using the programmatic mapping to build the model from the "injected" fields, but that didn't work. So instead, I've added this binder ^

[marko-bekhta]

the binder takes the binder type from the annotation attribute and then goes through the fields from that type, and that's where the "second" processing method for the annotations is used since, at this point, we are working with the index elements and not the programmatic api, hence the "original" processing method won't apply here...

[marko-bekhta]

this is just an interesting thing to point out, as we don't really care about some things when building this model, we can skip reading them and just hardcode some values instead.