Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FMWK-278 Refactor configuration #684

Merged
merged 19 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<springdata.commons>3.2.1</springdata.commons>
<springdata.keyvalue>3.2.1</springdata.keyvalue>
<springdata.spring-boot>3.2.1</springdata.spring-boot>
<spring-boot-starter-test>3.2.0</spring-boot-starter-test>
<spring-cloud-starter-bootstrap>4.1.0</spring-cloud-starter-bootstrap>
<maven.javadoc.plugin>3.3.0</maven.javadoc.plugin>
Expand Down Expand Up @@ -183,6 +184,12 @@
<artifactId>spring-data-keyvalue</artifactId>
<version>${springdata.keyvalue}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot</artifactId>
<version>${springdata.spring-boot}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.aerospike</groupId>
<artifactId>aerospike-client</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion src/main/asciidoc/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ include::reference/template.adoc[]
include::reference/secondary-indexes.adoc[]
include::reference/indexed-annotation.adoc[]
include::reference/caching.adoc[]
include::reference/configure-data-settings.adoc[]
include::reference/configuration.adoc[]
include::spring-data-commons-docs/dependencies.adoc[]
include::spring-data-commons-docs/auditing.adoc[]

Expand Down
2 changes: 1 addition & 1 deletion src/main/asciidoc/reference/aerospike-object-mapping.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The following outlines what field will be mapped to the '_id' document field:

The following outlines what type of conversion, if any, will be done on the property mapped to the _id document field.

* By default, the type of the field annotated with `@id` is turned into a `String` to be stored in Aerospike database. If the original type cannot be persisted (see xref:#configure-data-settings.keep-original-key-types[keepOriginalKeyTypes] for details), it must be convertible to `String` and will be stored in the database as such, then converted back to the original type when the object is read. This is transparent to the application but needs to be considered if using external tools like `AQL` to view the data.
* By default, the type of the field annotated with `@id` is turned into a `String` to be stored in Aerospike database. If the original type cannot be persisted (see xref:#configuration.keep-original-key-types[keepOriginalKeyTypes] for details), it must be convertible to `String` and will be stored in the database as such, then converted back to the original type when the object is read. This is transparent to the application but needs to be considered if using external tools like `AQL` to view the data.
* If no field named "id" is present in the Java class then an implicit '_id' file will be generated by the driver but not mapped to a property or field of the Java class.

When querying and updating `AerospikeTemplate` will use the converter to handle conversions of the `Query` and `Update` objects that correspond to the above rules for saving documents so field names and types used in your queries will be able to match what is in your domain classes.
Expand Down
10 changes: 5 additions & 5 deletions src/main/asciidoc/reference/caching.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -206,17 +206,17 @@ public class UserController {

=== Add @EnableCaching

==== SimpleSpringbootAerospikeCacheApplication
==== SimpleSpringBootAerospikeCacheApplication

Add `@EnableCaching` to the class that contains the main method.

[source,java]
----
@EnableCaching
@SpringBootApplication
public class SimpleSpringbootAerospikeCacheApplication {
public class SimpleSpringBootAerospikeCacheApplication {
public static void main(String[] args) {
SpringApplication.run(SimpleSpringbootAerospikeCacheApplication.class, args);
SpringApplication.run(SimpleSpringBootAerospikeCacheApplication.class, args);
}
}
----
Expand Down Expand Up @@ -252,7 +252,7 @@ aql> select * from test
+-----+-----------+----------+-------------+-------------------------------------+
| @user_key | name | @_class | email | age |
+-----+-----------+----------+-------------+-------------------------------------+
| "1" | "guthrie" | "com.aerospike.cache.simplespringbootaerospikecache.objects.User" | "[email protected]" | 35 |
| "1" | "guthrie" | "com.aerospike.cache.simpleSpringBootAerospikeCache.objects.User" | "[email protected]" | 35 |
+-----+-----------+----------+-------------+-------------------------------------+
----

Expand Down Expand Up @@ -315,6 +315,6 @@ aql> select * from test
+-----+-----------+----------+-------------+-------------------------------------+
| @user_key | name | @_class | email | age |
+-----+-----------+----------+-------------+-------------------------------------+
| "1" | "jimmy page" | "com.aerospike.cache.simplespringbootaerospikecache.objects.User" | "[email protected]" | 77 |
| "1" | "jimmy page" | "com.aerospike.cache.simpleSpringBootAerospikeCache.objects.User" | "[email protected]" | 77 |
+-----+-----------+----------+-------------+-------------------------------------+
----
Original file line number Diff line number Diff line change
@@ -1,38 +1,19 @@
[[configure-data-settings]]
= Configure Data Settings

The `AbstractAerospikeDataConfiguration` class exposes a number of beans that Spring Data Aerospike uses internally. User can optionally override `configureDataSettings` method to customize configuration.
[[configuration]]
= Configuration

Configuration can be accessed via standard `application.properties`.
Here is an example:

[source,java]
[source,properties]
----
@EnableAerospikeRepositories(basePackageClasses = PersonRepository.class)
class ApplicationConfig extends AbstractAerospikeDataConfiguration {

@Override
protected Collection<Host> getHosts() {
return Collections.singleton(new Host("localhost", 3000));
}

@Override
protected String nameSpace() {
return "test";
}

@Override
protected void configureDataSettings(AerospikeDataSettings.AerospikeDataSettingsBuilder builder) {
builder.createIndexesOnStartup(true);
builder.indexCacheRefreshSeconds(3600);
builder.queryMaxRecords(10000L);
}
}
spring-data-aerospike.hosts=localhost:3000
spring-data-aerospike.namespace=TEST
----

[[configure-data-settings.parameters]]
[[configuration.parameters]]
== Configuration Parameters

[[configure-data-settings.scans-enabled]]
[[configuration.scans-enabled]]
=== scansEnabled

A scan can be an expensive operation as all records in the set must be read by the Aerospike server, and then the condition is applied to see if they match.
Expand All @@ -43,49 +24,49 @@ NOTE: Once this flag is enabled, scans run whenever needed with no warnings. Thi

*Default:* `false`.

[[configure-data-settings.send-key]]
[[configuration.send-key]]
=== sendKey

Send user defined key in addition to hash digest on both reads and writes. If the key is sent on write, it will be stored with the record on the server.

Default: `false` (do not send the user defined key).

[[configure-data-settings.create-indexes-on-startup]]
[[configuration.create-indexes-on-startup]]
=== createIndexesOnStartup

Create secondary indexes specified using `@Indexed` annotation on startup.

*Default*: `true`.

[[configure-data-settings.index-cache-refresh-frequency-seconds]]
[[configuration.index-cache-refresh-frequency-seconds]]
=== indexCacheRefreshSeconds

Automatically refresh indexes cache every <N> seconds.

*Default*: `3600`.

[[configure-data-settings.server-version-refresh-frequency-seconds]]
[[configuration.server-version-refresh-frequency-seconds]]
=== serverVersionRefreshSeconds

Automatically refresh cached server version every <N> seconds.

*Default*: `3600`.

[[configure-data-settings.query-max-records]]
[[configuration.query-max-records]]
=== queryMaxRecords

Limit amount of results returned by server. Non-positive value means no limit.

*Default*: `10 000`.

[[configure-data-settings.batch-write-size]]
[[configuration.batch-write-size]]
=== batchWriteSize

Maximum batch size for batch write operations. Non-positive value means no limit.

*Default*: `100`.

[[configure-data-settings.keep-original-key-types]]
[[configuration.keep-original-key-types]]
=== keepOriginalKeyTypes

Define how `@Id` fields (primary keys) and `Map` keys are stored in the Aerospike database: `false` - always as `String`, `true` - preserve original type if supported.
Expand Down
5 changes: 3 additions & 2 deletions src/main/asciidoc/reference/indexed-annotation.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -148,5 +148,6 @@ So the following will be true:
|a.dd.[1].{#0} |[mapKey("a"), mapKey("dd"),listIndex(1), mapRank(0)] |y
|===

NOTE: There are relevant configuration parameters in AerospikeDataSettings:
xref:#configure-data-settings.create-indexes-on-startup[create indexes on startup] and xref:#configure-data-settings.index-cache-refresh-frequency-seconds[indexes cache refresh frequency].
NOTE: There are relevant configuration parameters in application.properties:
xref:#configuration.create-indexes-on-startup[create indexes on startup] and
xref:#configuration.index-cache-refresh-frequency-seconds[indexes cache refresh frequency].
27 changes: 5 additions & 22 deletions src/main/asciidoc/reference/installation-and-usage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -50,31 +50,14 @@ implementation group: 'com.aerospike', name: 'spring-data-aerospike', version: '

== Connecting to Aerospike DB

Configuring Spring Data Aerospike is easy with the help of the `AbstractAerospikeDataConfiguration` class.
Configuring Spring Data Aerospike is easy, necessary configuration should be specified in `application.properties`.

[source, java]
[source, properties]
----
@Configuration
@EnableAerospikeRepositories(basePackageClasses = { PersonRepository.class})
public class AerospikeConfiguration extends AbstractAerospikeDataConfiguration {
@Override
protected Collection<Host> getHosts() {
return Collections.singleton(new Host("localhost", 3000));
}

@Override
protected String nameSpace() {
return "test";
}
}
spring-data-aerospike.hosts=localhost:3000
spring-data-aerospike.namespace=TEST
----

`@Configuration` tells Spring that this class contains configuration data, and `@EnableAerospikeRepositories` activates Aerospike repositories that can be used for data access. The parameter to this annotation tells Spring Data Aerospike where to look for the repositories. This can be a list of package names as strings using the `basePackages` value, or a list of classes through the `basePackageClass` value. If the latter is used (as in this example), the class is used to determine which package to scan, and all repositories in that package will be available for use. More details on repositories are below.

The primary information required by this configuration is how to connect to the cluster, provided through the `getHosts` and `nameSpace` calls.

NOTE: You can optionally override xref:#configure-data-settings[`configureDataSettings`] method of `AbstractAerospikeDataConfiguration` to customize your configuration.

== Creating Functionality

The base functionality is provided by `AerospikeRepository` interface.
Expand Down Expand Up @@ -156,6 +139,6 @@ aql> select * from test.Person where pk = "1"

NOTE: The fully qualified path of the class is listed in each record. This is needed to instantiate the class correctly, especially in cases when the compile-time type and runtime type of the object differ. For example, where a field is declared as a super class but the instantiated class is a subclass.

NOTE: By default, the type of the field annotated with `@id` is turned into a `String` to be stored in Aerospike database. If the original type cannot be persisted (see xref:#configure-data-settings.keep-original-key-types[keepOriginalKeyTypes] for details), it must be convertible to `String` and will be stored in the database as such, then converted back to the original type when the object is read. This is transparent to the application but needs to be considered if using external tools like `AQL` to view the data.
NOTE: By default, the type of the field annotated with `@id` is turned into a `String` to be stored in Aerospike database. If the original type cannot be persisted (see xref:#configuration.keep-original-key-types[keepOriginalKeyTypes] for details), it must be convertible to `String` and will be stored in the database as such, then converted back to the original type when the object is read. This is transparent to the application but needs to be considered if using external tools like `AQL` to view the data.


32 changes: 8 additions & 24 deletions src/main/asciidoc/reference/scan-operation.adoc
Original file line number Diff line number Diff line change
@@ -1,36 +1,20 @@
[[scan-operation]]
= Scan Operation

A scan can be an expensive operation as all records in the set must be read by the Aerospike server, and then the condition is applied to see if they match.
A scan can be an expensive operation as all records in the set must be read by the Aerospike server,
and then the condition is applied to see if they match.

Due to the cost of performing this operation, scans from Spring Data Aerospike are disabled by default.

== Enabling Scan

If the cost of the scans is acceptable to an organization, they can be enabled by setting xref:#configure-data-settings.scans-enabled[`scansEnabled`] to `true` in `AerospikeDataSettings`.
If the cost of the scans is acceptable to an organization, they can be enabled by setting
xref:#configuration.scans-enabled[`scansEnabled`] to `true` in `application.properties`.

One way to do this is to create a custom bean which overrides the default settings:

[source,java]
[source,properties]
----
@Configuration
@EnableAerospikeRepositories(basePackageClasses = { PersonRepository.class})
public class AerospikeConfiguration extends AbstractAerospikeDataConfiguration {
@Override
protected Collection<Host> getHosts() {
return Collections.singleton(new Host("localhost", 3000));
}

@Override
protected String nameSpace() {
return "test";
}

@Bean
public AerospikeDataSettings aerospikeDataSettings() {
return AerospikeDataSettings.builder().scansEnabled(true).build();
}
}
spring-data-aerospike.scans-enabled=true
----

NOTE: Once this flag is enabled, scans run whenever needed with no warnings. This may or may not be optimal in a particular use case.
NOTE: Once this flag is enabled, scans run whenever needed with no warnings.
This may or may not be optimal in a particular use case.
5 changes: 3 additions & 2 deletions src/main/asciidoc/reference/secondary-indexes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ public List<Person> findByFirstName(String firstName);
In this case `firstName` is not marked as `@Indexed`, so SpringData Aerospike is not instructed to create an index on it.
Hence, it will scan the repository (a costly operation that could be avoided by using an index).

NOTE: There are relevant configuration parameters in AerospikeDataSettings:
xref:#configure-data-settings.create-indexes-on-startup[create indexes on startup] and xref:#configure-data-settings.index-cache-refresh-frequency-seconds[indexes cache refresh frequency].
NOTE: There are relevant configuration parameters in `application.properties`:
xref:#configuration.create-indexes-on-startup[create indexes on startup] and
xref:#configuration.index-cache-refresh-frequency-seconds[indexes cache refresh frequency].

Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
import org.springframework.data.aerospike.query.cache.InternalIndexOperations;
import org.springframework.data.aerospike.server.version.ServerVersionSupport;

import static org.springframework.data.aerospike.utility.Utils.getNamespace;

@Slf4j
@Configuration
public abstract class AbstractAerospikeDataConfiguration extends AerospikeDataConfigurationSupport {
Expand All @@ -45,21 +47,22 @@ public AerospikeTemplate aerospikeTemplate(IAerospikeClient aerospikeClient,
AerospikeMappingContext aerospikeMappingContext,
AerospikeExceptionTranslator aerospikeExceptionTranslator,
QueryEngine queryEngine, IndexRefresher indexRefresher,
ServerVersionSupport serverVersionSupport) {
return new AerospikeTemplate(aerospikeClient, nameSpace(), mappingAerospikeConverter,
aerospikeMappingContext, aerospikeExceptionTranslator, queryEngine, indexRefresher, serverVersionSupport);
ServerVersionSupport serverVersionSupport, AerospikeSettings settings) {
return new AerospikeTemplate(aerospikeClient, getNamespace(settings.getNamespace(), nameSpace()),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only aerospikeSettings we need in template is currently namespace but it makes sense to pass the entire aerospikeSettings to the template for more modular code flow (in case we want to use an aerospike setting in the template), WDYT?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall agree, but it might be more suitable to do in a separate ticket.

mappingAerospikeConverter, aerospikeMappingContext, aerospikeExceptionTranslator, queryEngine,
indexRefresher, serverVersionSupport);
}

@Bean(name = "aerospikeQueryEngine")
public QueryEngine queryEngine(IAerospikeClient aerospikeClient,
StatementBuilder statementBuilder,
FilterExpressionsBuilder filterExpressionsBuilder) {
FilterExpressionsBuilder filterExpressionsBuilder, AerospikeSettings settings) {
QueryEngine queryEngine = new QueryEngine(aerospikeClient, statementBuilder, filterExpressionsBuilder);
boolean scansEnabled = aerospikeDataSettings().isScansEnabled();
log.debug("AerospikeDataSettings.scansEnabled: {}", scansEnabled);
boolean scansEnabled = settings.isScansEnabled();
log.debug("AerospikeSettings.scansEnabled: {}", scansEnabled);
queryEngine.setScansEnabled(scansEnabled);
long queryMaxRecords = aerospikeDataSettings().getQueryMaxRecords();
log.debug("AerospikeDataSettings.queryMaxRecords: {}", queryMaxRecords);
long queryMaxRecords = settings.getQueryMaxRecords();
log.debug("AerospikeSettings.queryMaxRecords: {}", queryMaxRecords);
queryEngine.setQueryMaxRecords(queryMaxRecords);
return queryEngine;
}
Expand All @@ -68,23 +71,22 @@ public QueryEngine queryEngine(IAerospikeClient aerospikeClient,
public AerospikePersistenceEntityIndexCreator aerospikePersistenceEntityIndexCreator(
ObjectProvider<AerospikeMappingContext> aerospikeMappingContext,
AerospikeIndexResolver aerospikeIndexResolver,
ObjectProvider<AerospikeTemplate> template)
{
boolean indexesOnStartup = aerospikeDataSettings().isCreateIndexesOnStartup();
log.debug("AerospikeDataSettings.indexesOnStartup: {}", indexesOnStartup);
ObjectProvider<AerospikeTemplate> template, AerospikeSettings settings) {
boolean indexesOnStartup = settings.isCreateIndexesOnStartup();
log.debug("AerospikeSettings.indexesOnStartup: {}", indexesOnStartup);
return new AerospikePersistenceEntityIndexCreator(aerospikeMappingContext, indexesOnStartup,
aerospikeIndexResolver, template);
}

@Bean(name = "aerospikeIndexRefresher")
public IndexRefresher indexRefresher(IAerospikeClient aerospikeClient, IndexesCacheUpdater indexesCacheUpdater,
ServerVersionSupport serverVersionSupport) {
ServerVersionSupport serverVersionSupport, AerospikeSettings settings) {
IndexRefresher refresher = new IndexRefresher(aerospikeClient, aerospikeClient.getInfoPolicyDefault(),
new InternalIndexOperations(new IndexInfoParser()), indexesCacheUpdater, serverVersionSupport);
refresher.refreshIndexes();
int refreshFrequency = aerospikeDataSettings().getIndexCacheRefreshSeconds();
int refreshFrequency = settings.getIndexCacheRefreshSeconds();
processCacheRefreshFrequency(refreshFrequency, refresher);
log.debug("AerospikeDataSettings.indexCacheRefreshSeconds: {}", refreshFrequency);
log.debug("AerospikeSettings.indexCacheRefreshSeconds: {}", refreshFrequency);
return refresher;
}

Expand Down
Loading
Loading