[Pull-based Ingestion] Support segment replication for pull-based ingestion #17359
Conversation
github-actions
bot
added
enhancement
|
❌ Gradle check result for 7a682ef: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
7a682ef to
cea42e1
Compare
varunbharadwaj
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
cwperks,
dblock,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami,
VachaShah and
jainankitk
as code owners
|
❌ Gradle check result for cea42e1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
varunbharadwaj
changed the title
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
|
❌ Gradle check result for 5db9a2a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
5db9a2a to
57b86ed
Compare
|
❌ Gradle check result for 57b86ed: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
57b86ed to
a2f9dc2
Compare
|
❌ Gradle check result for a2f9dc2: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
a2f9dc2 to
c5adb5e
Compare
|
❌ Gradle check result for c5adb5e: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
...n-kafka/src/internalClusterTest/java/org/opensearch/plugin/kafka/KafkaIngestionSegRepIT.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/engine/IngestionDeletionPolicy.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/engine/IngestionNRTReplicationEngine.java
Outdated
Show resolved
Hide resolved
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
c5adb5e to
2d4be95
Compare
|
❌ Gradle check result for 2d4be95: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
2d4be95 to
bc9716c
Compare
varunbharadwaj
changed the title
|
❌ Gradle check result for bc9716c: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
…ecovery Signed-off-by: Varun Bharadwaj <[email protected]>
varunbharadwaj
force-pushed
the
vb/segrep
branch
2 times, most recently
from
b758603 to
a7c7a99
Compare
|
❌ Gradle check result for a7c7a99: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Varun Bharadwaj <[email protected]>
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
a7c7a99 to
fae7e91
Compare
|
❌ Gradle check result for fae7e91: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
| * | ||
| * @opensearch.internal | ||
| */ | ||
| public interface EngineTranslogManager extends TranslogManager, Closeable { |
There was a problem hiding this comment.
It isn't very clear for implementors to choose between EngineTranslogManager and TranslogManager except for some additionally exposed methods. Is the rationale to avoid breaking a breaking change for existing implementors
Description
This PR is a follow up for pull-based-ingestion to support segment replication with remote store. The primary shard will ingest from the streaming source and replica shards will rely on segment replication.
This PR refactors IngestionEngine to inherit from InternalEngine to support replication, recovery and avoid duplicate code. Some of the changes required to support segRep and peer recovery are enhancing IngestionEngine to include required listeners, support working with NRTReplicationEngine, tracking latest index commits, prevent snapshotted index deletion, among many others. These changes are already available in InternalEngine, and can be reused by IngestionEngine after this change.
Integration tests are added to validate end-to-end pull-based ingestion with segment replication, peer recover, replica promotion and remote store.
Related Issues
Resolves #16929
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.