HSEARCH-4487 + HSEARCH-4980 Two fixes for Jakarta Batch Mass Indexing job: MySQL with jdbcFetchSize=Integer.MIN_VALUE + embedded ids #3754
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yrodiere
force-pushed
the
HSEARCH-4487
branch
2 times, most recently
from
43f8365 to
fe83edd
Compare
|
Ok, I had to push two updates:
|
marko-bekhta
approved these changes
Sep 28, 2023
...src/main/java/org/hibernate/search/mapper/orm/massindexing/MassIndexerFilteringTypeStep.java
Outdated
Show resolved
Hide resolved
yrodiere
force-pushed
the
HSEARCH-4487
branch
from
fe83edd to
95669a1
Compare
yrodiere
force-pushed
the
HSEARCH-4487
branch
5 times, most recently
from
c40f929 to
3c3613e
Compare
These tests shouldn't have been disabled in the first place.
…mples The documentation example was contrived (weird data). There is no need to use "e." everywhere, as conditions work fine without that prefix.
…loading in separate sessions
… mass indexing job No need for a full-blown session just to count entities or scroll on identifiers.
|
Kudos, SonarCloud Quality Gate passed! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For HSEARCH-4980, it's just that tests were disabled so the implementation was garbage and didn't even work. I would know, I committed that nonsense. We might want to backport the fix to 6.2, though I'm not sure it's easy.
For HSEARCH-4487, the problem was that we were using a scroll on entities. When using MySQL with jdbcFetchSize=Integer.MIN_VALUE (AKA "I mean scrolling for real, not loading the whole database in memory"), this is fundamentally incompatible with lazy loading (because scrolling basically takes over the whole connection and one can't use it for lazy loading anymore).
So, I solved this by not scrolling entities anymore: we're doing things like in the MassIndexer, with one session (connection) scrolling on IDs, and the other loading entities in batches.
This obviously changes the performance profile quite a lot, so we should brace for performance regressions.
This also implies some API changes as well: in particular, I changed the "custom HQL" feature to be closer to what we have in the MassIndexer, and removed the (now nonsensical)
sessionClearIntervalsetting.But AFAIK this is the only way to solve the problem.
I won't bother backporting HSEARCH-4487 to 6.2, as backporting without breaking changes would require too much work and duplication.