Implement ORDER BY BM25

[jbellis]

What is the issue

https://github.com/riptano/cndb/issues/11725

What does this PR fix and why was it fixed

...

Checklist before you submit for review

• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits

[jbellis]

(force push is identical code that CI ran previously, just cleaned up the history)

[michaeljmarshall]

Looks great! I'm really happy with how much this cleaned up the index based ordering logic in several classes.

Left a handful of minor comments/questions.

[michaeljmarshall]

Nice work! I left several minor comments and a few larger questions.

[michaeljmarshall]

What is the motivation for using intersection here as opposed to union? I know that our : operator does intersection, but I think that is the opposite of the typical expectation for analyzed text search. Since BM25 is its own thing, I wonder if we can break with our previous design decision and union these results. I am assuming that docs with more terms will naturally be higher in the list, anyway.

[jbellis]

(1) we're materializing the results into memory and there's no real way around that so union increases the risk that something explodes

(2) I'm relying on "Our experiments focused on conjunctive query evaluation, where the document must contain all query terms; previous work [1] has shown that this approach yields comparable end-to-end effectiveness to disjunctive query evaluation, but is faster." in https://dl.acm.org/doi/abs/10.1145/2600428.2609460 [the work cited is by one of the authors so I assume it's accurate]

[jbellis]

we're materializing the results into memory and there's no real way around that

actually we can avoid this (while still doing a single pass) if we materialize average document length in the statistics

[jbellis]

3 of the CI failures are straightforwardly false positives.

VectorHybridSearchTest failure looks suspicious

junit.framework.AssertionFailedError: Resource leaks were detected during this test. Add -Dcassandra.debugrefcount=true to analyze the leaks expected:<0> but was:<1>
	at org.apache.cassandra.index.sai.utils.ResourceLeakDetector.afterIfSuccessful(ResourceLeakDetector.java:79)
	at org.apache.cassandra.index.sai.utils.ResourceLeakDetector$1.afterIfSuccessful(ResourceLeakDetector.java:65)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)

But I think it's also a false positive:

1. Does not reproduce locally
2. Between last successful run and failed run is exactly one commit (3ad8ae2) which only changes MultipleColumnIndexTest.java and NativeIndexDDLTest.java

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-1434 rejected by Butler

---

2 new test failure(s) in 16 builds
See build details here

---

Found 2 new test failures

Test	Explanation	Branch history	Upstream history
...a,dataset=int,wide=true,scenario=SSTABLE_QUERY]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...map<time,time>,wide=false,scenario=MIXED_QUERY]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵

---

Found 71 known test failures

[sonarqubecloud]

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.5% Coverage on New Code
0.7% Duplication on New Code

See analysis details on SonarQube Cloud