Skip to content

Commit

Permalink
Reduce side of bulks when indexing in dev, tests and prod
Browse files Browse the repository at this point in the history
It seems documents have gotten bigger and are now saturating the
OpenSearch memory, which is rather restricted in our current setup,
leading to failures.
  • Loading branch information
yrodiere committed Nov 7, 2024
1 parent 5eb509c commit c24d180
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions src/main/resources/application.properties
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,10 @@ quarkus.hibernate-search-standalone.elasticsearch.version=opensearch:2.18
quarkus.elasticsearch.devservices.image-name=opensearch-custom:${maven.version.opensearch}
quarkus.elasticsearch.devservices.java-opts=${PROD_OPENSEARCH_JAVA_OPTS}
# Limit parallelism of indexing, because OpenSearch can only handle so many documents in its buffers.
# This leads to at most 8*20=160 documents being indexed in parallel, which should be plenty
# This leads to at most 8*12=96 documents being indexed in parallel, which should be plenty
# given how large our documents can be.
INDEXING_QUEUE_COUNT=8
INDEXING_BULK_SIZE=20
INDEXING_BULK_SIZE=12
quarkus.hibernate-search-standalone.elasticsearch.indexing.queue-count=${INDEXING_QUEUE_COUNT}
quarkus.hibernate-search-standalone.elasticsearch.indexing.max-bulk-size=${INDEXING_BULK_SIZE}
# We need to apply a custom OpenSearch mapping to exclude very large fields from the _source
Expand Down

0 comments on commit c24d180

Please sign in to comment.