Support extending NEST types by deriving and implementing properties

[russcam]

With the move to utf8json in #3493, the serialization of NEST types is strict, serializing only those properties specified on implemented interfaces in many places. For example, PropertyFormatter only serializes the I*Property interface members:

elasticsearch-net/src/Nest/Mapping/Types/PropertyFormatter.cs

Lines 97 to 192 in f3ec719

	public void Serialize(ref JsonWriter writer, IProperty value, IJsonFormatterResolver formatterResolver)
	{
	if (value == null)
	{
	writer.WriteNull();
	return;
	}
	switch (value.Type)
	{
	case "text":
	Serialize<ITextProperty>(ref writer, value, formatterResolver);
	break;
	case "keyword":
	Serialize<IKeywordProperty>(ref writer, value, formatterResolver);
	break;
	case "float":
	case "double":
	case "byte":
	case "short":
	case "integer":
	case "long":
	case "scaled_float":
	case "half_float":
	Serialize<INumberProperty>(ref writer, value, formatterResolver);
	break;
	case "date":
	Serialize<IDateProperty>(ref writer, value, formatterResolver);
	break;
	case "boolean":
	Serialize<IBooleanProperty>(ref writer, value, formatterResolver);
	break;
	case "binary":
	Serialize<IBinaryProperty>(ref writer, value, formatterResolver);
	break;
	case "object":
	Serialize<IObjectProperty>(ref writer, value, formatterResolver);
	break;
	case "nested":
	Serialize<INestedProperty>(ref writer, value, formatterResolver);
	break;
	case "ip":
	Serialize<IIpProperty>(ref writer, value, formatterResolver);
	break;
	case "geo_point":
	Serialize<IGeoPointProperty>(ref writer, value, formatterResolver);
	break;
	case "geo_shape":
	Serialize<IGeoShapeProperty>(ref writer, value, formatterResolver);
	break;
	case "completion":
	Serialize<ICompletionProperty>(ref writer, value, formatterResolver);
	break;
	case "token_count":
	Serialize<ITokenCountProperty>(ref writer, value, formatterResolver);
	break;
	case "murmur3":
	Serialize<IMurmur3HashProperty>(ref writer, value, formatterResolver);
	break;
	case "percolator":
	Serialize<IPercolatorProperty>(ref writer, value, formatterResolver);
	break;
	case "date_range":
	Serialize<IDateRangeProperty>(ref writer, value, formatterResolver);
	break;
	case "double_range":
	Serialize<IDoubleRangeProperty>(ref writer, value, formatterResolver);
	break;
	case "float_range":
	Serialize<IFloatRangeProperty>(ref writer, value, formatterResolver);
	break;
	case "integer_range":
	Serialize<IIntegerRangeProperty>(ref writer, value, formatterResolver);
	break;
	case "long_range":
	Serialize<ILongRangeProperty>(ref writer, value, formatterResolver);
	break;
	case "ip_range":
	Serialize<IIpRangeProperty>(ref writer, value, formatterResolver);
	break;
	case "join":
	Serialize<IJoinProperty>(ref writer, value, formatterResolver);
	break;
	case "alias":
	Serialize<IFieldAliasProperty>(ref writer, value, formatterResolver);
	break;
	default:
	if (value is IGenericProperty genericProperty)
	Serialize<IGenericProperty>(ref writer, genericProperty, formatterResolver);
	else
	{
	var formatter = DynamicObjectResolver.ExcludeNullCamelCase.GetFormatter<IProperty>();
	formatter.Serialize(ref writer, value, formatterResolver);
	}
	break;
	}

With such strictness, it is not currently possible to derive a type from a NEST type, add additional properties, and have those properties serialized by the internal serializer. This is demonstrated in InjectACustomIPropertyImplementation()

elasticsearch-net/src/Tests/Tests/ClientConcepts/HighLevel/Serialization/ExtendingNestTypes.doc.cs

Lines 53 to 97 in e2b7f75

	[U (Skip = "TODO: Does not work with utf8json")]
	public void InjectACustomIPropertyImplementation()
	{
	/**
	* `PropertyNameAttribute` can be used to mark properties that should be serialized. Without this attribute,
	* NEST won't pick up the property for serialization.
	*
	* Now that we have our own `IProperty` implementation we can add it to our propertes mapping when creating an index
	*/
	var createIndexResponse = _client.CreateIndex("myindex", c => c
	.Map<Project>(m => m
	.Properties(props => props
	.Custom(new MyPluginProperty("fieldName", "dutch"))
	)
	)
	);
	/**
	* which will serialize to the following JSON request
	*/
	// json
	var expected = new
	{
	mappings = new
	{
	properties = new
	{
	fieldName = new
	{
	type = "my_plugin_property",
	language = "dutch",
	numeric = true
	}
	}
	}
	};
	/**
	* Whilst NEST can _serialize_ our `my_plugin_property`, it does not know how to _deserialize_ it;
	* We plan to make this more pluggable in the future
	*/
	// hide
	Expect(expected).FromRequest(createIndexResponse);
	}
	}

utf8json is able to support this feature through using the fallback formatter, DynamicObjectTypeFallbackFormatter ,exposed by resolving a IJsonFormatter<object>, which internally inspects the type and generates a custom serialization method for it, caching it in a ThreadsafeTypeKeyHashTable<KeyValuePair<object, SerializeMethod>>. This path looks like it would be a plausible way to support extending NEST types but requires some guarding when serializing Attribute types; when a type to serialize is an Attribute type, the walking of properties performed by MetaType traverses down to a framework enum type with multiple members with the same undelying value, which trips up the EnumFormatter. For an Attribute type, this walking should not be performed.

[joostPieterse]

We are currently running into this issue: we have a custom token filter that has an extra string property, which does not get serialized by the new serializer.
Is there currently any way we can get Nest to serialize this property?

[russcam]

Not with the high level client call at the moment, @joostPieterse. It's still possible to send the request as an anonymous type with the low level client, although I appreciate it is more work.

With the change in #4032, it will be good to revisit this. to see if we can make this easier in the high level client again.

[bonny-bonev]

We are experiencing similar problem. We index documents, that implement ICustomTypeDescriptor, but since the utf8json upgrade, the serializer is not respecting it. It just serializes the public properties, which generates completely wrong json and causes the index call to fail. Do you have any plans on fixing this? Before the utf8json upgrade it worked perfectly.

[bonny-bonev]

We upgraded to 7.5 (latest at this point), but sadly the problem is still present. We have an object, implementing ICustomTypeDescriptor, but the properties sent by the ElasticClient are the ones from the object type, not the object itself. We need this object to have dynamic properties. Is there a workaround for this?
P.S. If you need any additional information, we can provide it.

[russcam]

@bonny-bonev would you be able to provide some more detail on which types you're deriving from? Might be easiest to see what you're doing in a prior NEST version that works

[bonny-bonev]

@russcam - In the previous version we used (5.5) this worked perfectly. We have an object, implementing ICustomTypeDescriptor and extending DynamicObject, but if this is not supported, we can implement any other interface that is used to retrieve the properties of our object. Basically we have an object, holding a dictionary and returning some or all of the values from the dictionary as it's properties using the mentioned ICustomTypeDescriptor interface. After the serializer change in Elasticsearch to utf8json this stopped working and the client is getting all of the object properties disregarding the ICustomTypeDescriptor interface. We can implement any interface, or use a marker attribute, but I can't find any information on how to do this.

[russcam]

It's probably easier to see what you have in code @bonny-bonev. Would you be able to provide a succinct, but complete, example? Specifically, I'm interested in seeing which NEST types are derived from, so that I can investigate a way forward.

[bonny-bonev]

We are not inheriting any NEST type. The object we are sending looks something like this:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Dynamic;
using System.Linq;
public class MyCustomDocument : DynamicObject, ICustomTypeDescriptor

    {
        public string TestProperty {get; set;}

        public AttributeCollection GetAttributes()
        {
            return new AttributeCollection(null);
        }

        public string GetClassName()
        {
            return null;
        }

        public string GetComponentName()
        {
            return null;
        }

        public TypeConverter GetConverter()
        {
            return null;
        }

        public EventDescriptor GetDefaultEvent()
        {
            return null;
        }

        public PropertyDescriptor GetDefaultProperty()
        {
            return null;
        }

        public object GetEditor(Type editorBaseType)
        {
            return null;
        }

        public EventDescriptorCollection GetEvents(Attribute[] attributes)
        {
            return new EventDescriptorCollection(null);
        }

        public EventDescriptorCollection GetEvents()
        {
            return new EventDescriptorCollection(null);
        }

        public PropertyDescriptorCollection GetProperties(Attribute[] attributes)
        {
            return this.GetPropertyDescriptorCollection();
        }

        public PropertyDescriptorCollection GetProperties()
        {
            return ((ICustomTypeDescriptor)this).GetProperties(null);
        }

        public object GetPropertyOwner(PropertyDescriptor pd)
        {
            return this;
        }

        private PropertyDescriptorCollection GetPropertyDescriptorCollection()
        {
            var propertyNames = new List<string>() { "propName" }; // There is custom mechanism for getting the current properties for the document, but for the example static list is sufficient.
            var properties = new PropertyDescriptor[propertyNames.Count];
            for (int i = 0; i < propertyNames.Count; i++)
            {
                // We use custom property descriptor, but just comment this line. The idea is that the ElasticSearch client is not even calling this method to get the object properties.
                // We need to have extensibility to specify the object properties dynamically for the specific instance that we are sending for indexing.
                properties[i] = new CustomPropertyDescriptor(propertyNames[i]);
            }

            return new PropertyDescriptorCollection(properties);
        }
    }

The problem is that the client is not indexing the dynamic properties of the object, but just the static properties of the class "MyCustomDocument".
Do you understand what is the issue, or you need more details. I will assist any way I can. I am not sending the entire code, because it involves many classes not related to the problem.
We need to index instances of "MyCustomDocument" where every instance have different properties. In the example, the "propName" property is never retrieved, but the "TestProperty" is serialized. We don't want to serialize the "TestProperty", but only the dynamically included "propName".
Before the upgrade we used version "5.5" and the properties were serialized correctly and the "GetProperties" method was called correctly.

[russcam]

OK, I think I see what the problem is here, @bonny-bonev. This is quite a different issue to the one to which this issue relates, though; the issue you describe is related to documents you control that you need to index, whereas this issue is related to deriving types from NEST types in order to include additional properties.

It looks to me like MyCustomDocument is using features of Json.NET that understand how to serialize a type that implements ICustomTypeDescriptor. You should be able to achieve the same thing in 7.x by using Json.NET as the source serializer:

var defaultIndex = "default_index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool, JsonNetSerializer.Default)
    .DefaultIndex(defaultIndex);

var client = new ElasticClient(settings);

I tried to write an example to demonstrate, but I don't know what CustomPropertyDescriptor looks like, or how an instance of MyCustomDocument should be constructed.

[bonny-bonev]

Thank you. There is no need to write example, I will try this and get back to you. Thanks again for your time - it seems that you understand the issue, so I hope the solution will work.

[bonny-bonev]

Hi again. The solution you suggested seems to work for us at this point. Thank you very much!

[russcam]

I'm going to close this issue and open a new specifically to track extending token filters #3655 (comment)