diff --git a/crates/iota-data-ingestion-core/README.md b/crates/iota-data-ingestion-core/README.md index 6b244703255..4a4ce223c36 100644 --- a/crates/iota-data-ingestion-core/README.md +++ b/crates/iota-data-ingestion-core/README.md @@ -165,7 +165,7 @@ pub struct WorkerPool { // An unique name of the Worker task pub task_name: String, // How many instances of the current Worker to create, more workers are created - // more checkpoints they can process in parallel + // more checkpoints they can process concurrently concurrency: usize, // The actual Worker instance itself worker: Arc, @@ -191,7 +191,7 @@ The worker pool's size is determined by the `concurrency` value. The pool is imp } ``` -- The first branch first marks the `worker_id` as idle updates the watermark, and enforces ordered checkpoint execution. It then checks the cache for pending checkpoints and assigns any found to the idle `worker_id`. +- The first branch first marks the `worker_id` as idle, updates the watermark, and enforces ordered checkpoint execution. It then checks the cache for pending checkpoints and assigns any found to the idle `worker_id`. - The second branch handles incoming `CheckpointData` from the `CheckpointReader`. It discards checkpoints with sequence numbers less than the `current_checkpoint_number`, assuming they have already been processed. It then invokes the optional `preprocess_hook` method (defined in the `Worker` trait) to allow for validation, preparation, or caching of checkpoint data. If the hook executes successfully and an idle worker is available, the checkpoint is sent for processing otherwise, it is added to the cache for later processing. ### Indexer Executor @@ -202,13 +202,14 @@ The `IndexerExecutor` actor is the coordinator of the entire framework logic. pub struct IndexerExecutor

{ // Holds the registered WorkerPools actors pools: Vec + Send>>>, - // Store the Sender half of the channel to notofy Worker Poolf of new CheckpointData + // Store the Sender half of the channel to notofy Worker Pool of new CheckpointData pool_senders: Vec>, // A wrapper around the implemented ProgressStore by having an internal cache progress_store: ProgressStoreWrapper

, - // Wroker Pools will send on this channel and notify the Executor that the Checkpoint was synced and a GC operation is needed + // Worker Pools will send on this channel and notify the Executor that + // the Checkpoint was synced and a GC operation can be performed pool_progress_sender: mpsc::Sender<(String, CheckpointSequenceNumber)>, - // Listens on Synced checkpoints from Worker Pools and performs GC operation + // Listens on synced checkpoints from Worker Pools and performs GC operation pool_progress_receiver: mpsc::Receiver<(String, CheckpointSequenceNumber)>, metrics: DataIngestionMetrics, }