Skip to content

Commit

Permalink
fixup! fixup! fixup! fixup! feat(iota-data-ingestion-core): add READM…
Browse files Browse the repository at this point in the history
…E.md
  • Loading branch information
sergiupopescu199 committed Dec 20, 2024
1 parent ddd495e commit 8ab3576
Showing 1 changed file with 6 additions and 5 deletions.
11 changes: 6 additions & 5 deletions crates/iota-data-ingestion-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ pub struct WorkerPool<W: Worker> {
// An unique name of the Worker task
pub task_name: String,
// How many instances of the current Worker to create, more workers are created
// more checkpoints they can process in parallel
// more checkpoints they can process concurrently
concurrency: usize,
// The actual Worker instance itself
worker: Arc<W>,
Expand All @@ -191,7 +191,7 @@ The worker pool's size is determined by the `concurrency` value. The pool is imp
}
```

- The first branch first marks the `worker_id` as idle updates the watermark, and enforces ordered checkpoint execution. It then checks the cache for pending checkpoints and assigns any found to the idle `worker_id`.
- The first branch first marks the `worker_id` as idle, updates the watermark, and enforces ordered checkpoint execution. It then checks the cache for pending checkpoints and assigns any found to the idle `worker_id`.
- The second branch handles incoming `CheckpointData` from the `CheckpointReader`. It discards checkpoints with sequence numbers less than the `current_checkpoint_number`, assuming they have already been processed. It then invokes the optional `preprocess_hook` method (defined in the `Worker` trait) to allow for validation, preparation, or caching of checkpoint data. If the hook executes successfully and an idle worker is available, the checkpoint is sent for processing otherwise, it is added to the cache for later processing.

### Indexer Executor
Expand All @@ -202,13 +202,14 @@ The `IndexerExecutor` actor is the coordinator of the entire framework logic.
pub struct IndexerExecutor<P> {
// Holds the registered WorkerPools actors
pools: Vec<Pin<Box<dyn Future<Output = ()> + Send>>>,
// Store the Sender half of the channel to notofy Worker Poolf of new CheckpointData
// Store the Sender half of the channel to notofy Worker Pool of new CheckpointData
pool_senders: Vec<mpsc::Sender<CheckpointData>>,
// A wrapper around the implemented ProgressStore by having an internal cache
progress_store: ProgressStoreWrapper<P>,
// Wroker Pools will send on this channel and notify the Executor that the Checkpoint was synced and a GC operation is needed
// Worker Pools will send on this channel and notify the Executor that
// the Checkpoint was synced and a GC operation can be performed
pool_progress_sender: mpsc::Sender<(String, CheckpointSequenceNumber)>,
// Listens on Synced checkpoints from Worker Pools and performs GC operation
// Listens on synced checkpoints from Worker Pools and performs GC operation
pool_progress_receiver: mpsc::Receiver<(String, CheckpointSequenceNumber)>,
metrics: DataIngestionMetrics,
}
Expand Down

0 comments on commit 8ab3576

Please sign in to comment.