NullPointer exception when overriding AerospikeTypeAliasAccessor(null) after update to 4.7.1

[Ferioney]

Hi folks,

I upgraded my application from Spring Data Aerospike from 3.5.0 to 4.7.1.
In the application, I override AerospikeTypeAliasAccessor bean to skip writing @class bin:

@Bean
@Override
public AerospikeTypeAliasAccessor aerospikeTypeAliasAccessor() {
    //we do not want to save @class field with type of document into Aerospike
    return new AerospikeTypeAliasAccessor(null);
}

My application does regular findById and update (save) operations.

Then I deployed my application to the environment. I use k8s rolling update. After a new pod (based on 4.7.1) started I saw that all older pods (based on 3.5.0) threw a null pointer exception for findById operation:

NullPointerException log

	java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:233)
	at java.base/java.util.TreeMap.getEntry(TreeMap.java:380)
	at java.base/java.util.TreeMap.get(TreeMap.java:290)
	at org.springframework.data.aerospike.convert.AerospikeTypeAliasAccessor.readAliasFrom(AerospikeTypeAliasAccessor.java:38)
	at org.springframework.data.aerospike.convert.AerospikeTypeAliasAccessor.readAliasFrom(AerospikeTypeAliasAccessor.java:23)
	at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:120)
	at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:182)
	at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:152)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter.convertCustomType(MappingAerospikeReadConverter.java:161)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter.readValue(MappingAerospikeReadConverter.java:147)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter.access$300(MappingAerospikeReadConverter.java:51)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter$RecordReadingPropertyValueProvider.getPropertyValue(MappingAerospikeReadConverter.java:278)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter.lambda$convertProperties$0(MappingAerospikeReadConverter.java:113)
	at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:368)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter.convertProperties(MappingAerospikeReadConverter.java:105)
	at org.springframework.data.aerospike.convert.MappingAerospikeReadConverter.read(MappingAerospikeReadConverter.java:90)
	at org.springframework.data.aerospike.convert.MappingAerospikeConverter.read(MappingAerospikeConverter.java:75)
	at org.springframework.data.aerospike.core.BaseAerospikeTemplate.mapToEntity(BaseAerospikeTemplate.java:130)
	at org.springframework.data.aerospike.core.AerospikeTemplate.getRecordMapToEntityClass(AerospikeTemplate.java:364)
	at org.springframework.data.aerospike.core.AerospikeTemplate.findByIdInternal(AerospikeTemplate.java:336)
	at org.springframework.data.aerospike.core.AerospikeTemplate.findById(AerospikeTemplate.java:315)
	at org.springframework.data.aerospike.repository.support.SimpleAerospikeRepository.findById(SimpleAerospikeRepository.java:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:289)
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
	at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:530)
	at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:286)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:640)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:164)
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:139)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

After research, I found that version 4.2.0 changed the behaviour of map-key ordering (in the case of the ordered map TreeMap is used, FMWK-53 Find by POJO). These changes were not applicable when using AerospikeTypeAliasAccessor(null) Fix null pointer issue when typeKey is set as null and it was fixed in 4.7.0.

Now I have a situation where some documents that were updated by the new pods (4.7.1) can not be fetched by older pods (3.5.0).

I understand that I can deploy a new version with the new map-key ordering and wait until older pods are killed. But for deploying
time I will have a lot of errors. In a real production environment, I have а hundreds pods of applications and high load. Deploying time can take 30-60 mins. In case of deploying, I will affect real users for deploying time.

I didn't find any possibility to change map-key ordering. Is it possible to add some flag that can switch map-key ordering?

[agrgr]

Hello @Ferioney,
Thanks for the inquiry.
Theoretically we can release version 3.5.1 with an added null check in AliasAccessor so that Maps written with newer versions could be read without NPE if typeKey == null. Upgrade process will then consist of firstly migrating to 3.5.1 and then to any newer version.
Will it suit your situation?

[vladislav-sidorovich]

Hello,
the proposed solution doesn't look suitable. We have 500+ microservices and the update in 2 steps doesn't look realistic.

Could you please support backward compatibility in your clients? Once backward compatibility will be provided from aerospike client version x.y.z, we will block internally all the clients with the lower versions. Such an approach will help us to keep the production environment healthy.

Thanks

[reugn]

@vladislav-sidorovich, could you elaborate on your client backward compatibility request?

[agrgr]

@vladislav-sidorovich Can you please elaborate on what is exactly meant by "backward compatibility in aerospike clients"?
Spring Data Aerospike uses Java client which is not limited by frameworks.

The change of writing Maps to TreeMaps instead of HashMaps has been applied in Spring Data Aerospike and across different Aerospike frameworks as far as I remember to ensure comparing Maps.

As an alternative solution, there is an opportunity to add a configuration flag that will enable to write Maps as HashMaps instead of TreeMaps.
It will be possible to use the flag during upgrade and then set its value back to default after upgrade is over. The flag can be added to the upcoming release that is expected within couple weeks.
Does it suit better?

[vladislav-sidorovich]

@agrgr yes, the flag is much better, we will use the flag during the library update and it should help us.

@reugn the scenario is the following:
There is a cluster of N nodes of some Java service with aerospike-client 3.5.0. The new version of this Java service has aerospike-client 4.7.1. We start rolling out a new version of Java service to production and in some period of time there are N Java services with previous client and K services with the new version. Eventually, N will be reduced to 0, and K will be increased to N.
But during the rollout, the previous version of the service (with aerospkie-client 3.5.0) is not able to read data from Aerospike written by the new service version (with aerospike-client 4.7.1).

[reugn]

I see, so by "client" you meant the Spring Data library.

[vladislav-sidorovich]

Yes, sorry if it was confusing. I mean spring-data-aerospike, you can find pointers to the code in the initial post.

[agrgr]

The configuration flag will be a part of the upcoming 4.8.0 release expected within about a week.

[agrgr]

@Ferioney and @vladislav-sidorovich, the version 4.8.0 with the configuration flag has been released.

[reugn]

@Ferioney, @vladislav-sidorovich PR #745 introduced a configuration property that allows using unsorted maps for write operations, which provides backward compatibility with previous library versions. Please note that we recommend switching to the default after the migration is complete.
For more information about the configuration options, seek the writeSortedMaps property on the documentation page.

[Ferioney]

@agrgr @reugn Thank you!
I will check with the new property

[reugn]

@Ferioney feel free to reopen if you have any further questions.