feat: overhaul task spawning in the whole project #433

armyhaylenko · 2025-02-27T16:10:13Z

Overhaul task spawning in all modules. Remove usage of the mutexed_tasks concept, remove usage of a broadcast channel that was used for shutdown, create a static executor to save running tasks & exit gracefully.

kstepanovdev · 2025-02-28T11:14:51Z

Cargo.toml

@@ -20,7 +20,7 @@ members = [
 # Tokio ecosystem
 tokio = { version = "1.37.0", features = ["full", "tracing"] }
 tokio-stream = "0.1.11"
-tokio-util = { version = "0.7.10", features = ["codec", "compat"] }
+tokio-util = { version = "0.7.13", features = ["codec", "compat"] }


Do we actually need this patch version?

yes - it introduces a function called run_until_cancelled, which we use currently in one place, but the usage can be extended to other places. this PR is already destructive as it is, but run_until_cancelled can very easily replace this construct:

tokio::select! { _ = cancellation_token.cancelled() => {} _ = other_async_action_like_a_long_sleep => {} }

Maybe 0.7 will do the trick as well then? Instead of juggling patch versions all the time

kstepanovdev · 2025-02-28T11:34:08Z

consistency_check/src/bin/compressed_assets/main.rs

-            if !shutdown_for_file_writer_rx.is_empty() {
-                break;
-            }
+                if cancellation_token.is_cancelled() {


Does it mean that if cancellation is received during sleep or proccess_failed_proofs (which can take quite a long time I presume) we wait for their completion?

yes, if we start the iteration it will complete at least once. that's the logic that was here before and i don't find it to be erroneous

kstepanovdev · 2025-02-28T11:37:20Z

consistency_check/src/bin/jsons/main.rs

-            f_ch.clear();
-
-            drop(f_ch);
+                if cancellation_token.is_cancelled() {


The same question as above. Does it mean all write_record happen anyway even if shutdown signal is received?

yes, as it did before

kstepanovdev · 2025-02-28T11:40:59Z

nft_ingester/src/api/backfilling_state_consistency.rs

+            usecase::executor::spawn(async move {
+                while !cancellation_token.is_cancelled() {
+                    // TODO: refactor this to use parameter from storage and last slot from slot storage
+                    // overwhelm_backfill_gap.store(
+                    //     rocks_db.bubblegum_slots.iter_start().count().saturating_add(rocks_db.ingestable_slots.iter_start().count())
+                    //         >= consistence_backfilling_slots_threshold as usize,
+                    //     Ordering::Relaxed,
+                    // );
+                    //
+                    tokio::time::sleep(Duration::from_secs(CATCH_UP_SEQUENCES_TIMEOUT_SEC)).await;


What does it do except for sleeping?

nothing - as it did before

kstepanovdev · 2025-02-28T11:46:00Z

nft_ingester/src/backfiller.rs

@@ -132,13 +132,13 @@ pub async fn run_backfill_slots<C>(
    C: BlockConsumer,
 {
    loop {
-        if shutdown_token.is_cancelled() {
+        if cancellation_token.is_cancelled() {


May shutdown be blocked for 400ms? Is it possible to select the whole function, e.g. select over timer and token and when the timer ticks we execute its body or something like that?

400ms is not exactly a long time and we can easily wait IMO

yes but this PR is big already. i'm happy to do it in a different PR

kstepanovdev · 2025-02-28T11:48:11Z

nft_ingester/src/backfiller.rs

@@ -182,7 +182,7 @@ where
        it.seek_to_first();
    }
    while it.valid() {
-        if shutdown_token.is_cancelled() {
+        if cancellation_token.is_cancelled() {


The same concern. It's not a problem in general if the rest of the body will take a short time to complete because then we will guarantee that a message is processed and not lost though.

each iteration here seems to be complete, we don't batch it or whatever; i think what we had before is fine

kstepanovdev · 2025-02-28T11:49:22Z

nft_ingester/src/batch_mint/batch_mint_persister.rs

+        &self,
+        cancellation_token: CancellationToken,
+    ) -> Result<(), JoinError> {
+        while !cancellation_token.is_cancelled() {


get_batch_mint_to_verify will block the shutdown

yes - it's graceful and we will complete existing work and won't start new iterations or whatever

kstepanovdev · 2025-02-28T11:50:24Z

nft_ingester/src/batch_mint/batch_mint_processor.rs

-    pub async fn process_batch_mints(&self, mut rx: Receiver<()>) {
-        while rx.is_empty() {
+    pub async fn process_batch_mints(&self, cancellation_token: CancellationToken) {
+        while !cancellation_token.is_cancelled() {


shutdown will be blocked by fetch_batch_mint_for_processing, is it intended?

as it was intended before; only the current iteration will be blocked

kstepanovdev · 2025-02-28T11:51:19Z

nft_ingester/src/batch_mint/batch_mint_processor.rs

        mut batch_mint_to_process: BatchMintWithState,
    ) -> Result<(), IngesterError> {
        info!("Processing {} batch_mint file", &batch_mint_to_process.file_name);
        let start_time = Instant::now();
        let (batch_mint, file_size, file_checksum) =
            self.read_batch_mint_file(&batch_mint_to_process).await?;
        let mut metadata_url = String::new();
-        while rx.is_empty() {
+        while !cancellation_token.is_cancelled() {


inside the match shutdown is impossible.

i view it as do {} while, we do the current stuff and then exit

kstepanovdev · 2025-02-28T11:54:50Z

nft_ingester/src/bin/backfill/main.rs

+            let cancellation_token = cancellation_token.child_token();
+            async move {
+                while let Ok((slot, raw_block_data)) = slot_receiver.recv().await {
+                    if cancellation_token.is_cancelled() {


During the body execution shutting down is impossible

it's fine because we want to finish the work we started

kstepanovdev · 2025-02-28T11:56:34Z

nft_ingester/src/bin/backfill/main.rs

+
+                    // Increment slots_processed
+                    let current_slots_processed =
+                        slots_processed.fetch_add(1, Ordering::Relaxed) + 1;


That +1 outside of atomic operation looks unusual to me. Why not +2 then inside the fetch_add?

fetch_add returns the previous value (before the increment), so to get the value after the call we have to add 1

It might overflow the current_slots_processed

pretty sure this won't be the case in the coming years

kstepanovdev · 2025-02-28T11:57:46Z

nft_ingester/src/bin/backfill/main.rs


-        loop {
-            tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+                if cancellation_token.is_cancelled() {


The body is blocked. Is not it possible to use select! for such cases?

this won't work for too long and IMO is fine

andrii-kl · 2025-02-28T12:01:29Z

I would add an additional function for graceful shutdown of the PG connections pool since it is not implemented now, so there may be connection leaks or data inconsistency on long queries. pool.close().await;

andrii-kl · 2025-02-27T19:13:03Z

consistency_check/src/bin/jsons/main.rs

-        let missed_jsons_file =
-            File::create(MISSED_JSONS_FILE).expect("Failed to create file for missed jsons");
+    usecase::executor::spawn({
+        let cancellation_token = cancellation_token.child_token();


It might be worth to check the token before we start processing?

not in the utility binaries IMO

andrii-kl · 2025-02-28T11:12:41Z

consistency_check/src/bin/compressed_assets/main.rs

-                error!("Error writing failed checks: {}", e);
-            }
+    usecase::executor::spawn({
+        let cancellation_token = cancellation_token.child_token();


the same, may to check parent cancellation before start data processing.

not in the utility binaries IMO

kstepanovdev · 2025-02-28T12:02:21Z

nft_ingester/src/bin/backfill/main.rs

+                // Update rate
+                {
+                    let mut rate_guard = rate.lock().unwrap();
+                    *rate_guard = current_rate;
+                }


Suggested change

// Update rate

{

let mut rate_guard = rate.lock().unwrap();

*rate_guard = current_rate;

}

*rate.lock().unwrap() = current_rate;

Isn't it possible? Why do we need a dedicated block for it? I guess it will be freed at the end of the loop, won't it?

this change is out of scope for this PR. i will create a task gathering such places and will submit the refactoring changes in a later PR

kstepanovdev · 2025-02-28T12:05:30Z

nft_ingester/src/bin/ingester/main.rs

-        while first_processed_slot.load(Ordering::Relaxed) == 0 && shutdown_rx.is_empty() {
-            tokio_sleep(Duration::from_millis(100)).await
+        while first_processed_slot.load(Ordering::Relaxed) == 0
+            && !cancellation_token.is_cancelled()


From my perspective, the cancellation token does nothing here.

it does - we will exit if the first processed slot is still 0 and SIGINT is triggered

kstepanovdev · 2025-02-28T12:10:23Z

nft_ingester/src/bin/ingester/main.rs

+        usecase::executor::spawn({
+            let cancellation_token = cancellation_token.child_token();
+            async move {
+                let program_id = mpl_bubblegum::programs::MPL_BUBBLEGUM_ID;
+                while !cancellation_token.is_cancelled() {
+                    match signature_fetcher
+                        .fetch_signatures(program_id, args.rpc_retry_interval_millis)
+                        .await
+                    {
+                        Ok(_) => {
+                            metrics_clone.inc_run_fetch_signatures(
+                                "fetch_signatures",
+                                MetricStatus::SUCCESS,
+                            );
+                            info!(
+                                "signatures sync finished successfully for program_id: {}",
+                                program_id
+                            );
+                        },
+                        Err(e) => {
+                            metrics_clone.inc_run_fetch_signatures(
+                                "fetch_signatures",
+                                MetricStatus::FAILURE,
+                            );
+                            error!(
+                                "signatures sync failed: {:?} for program_id: {}",
+                                e, program_id
+                            );
+                        },
+                    }

-                tokio_sleep(Duration::from_secs(60)).await;
+                    tokio::select! {
+                        _ = cancellation_token.cancelled() => {}
+                        _ = tokio::time::sleep(Duration::from_secs(60)) => {}
+                    }
+                }
            }
-
-            Ok(())
        });


Suggested change

usecase::executor::spawn({

let cancellation_token = cancellation_token.child_token();

async move {

let program_id = mpl_bubblegum::programs::MPL_BUBBLEGUM_ID;

while !cancellation_token.is_cancelled() {

match signature_fetcher

.fetch_signatures(program_id, args.rpc_retry_interval_millis)

.await

{

Ok(_) => {

metrics_clone.inc_run_fetch_signatures(

"fetch_signatures",

MetricStatus::SUCCESS,

);

info!(

"signatures sync finished successfully for program_id: {}",

program_id

);

},

Err(e) => {

metrics_clone.inc_run_fetch_signatures(

"fetch_signatures",

MetricStatus::FAILURE,

);

error!(

"signatures sync failed: {:?} for program_id: {}",

e, program_id

);

},

}

tokio_sleep(Duration::from_secs(60)).await;

tokio::select! {

_ = cancellation_token.cancelled() => {}

_ = tokio::time::sleep(Duration::from_secs(60)) => {}

}

}

}

Ok(())

});

usecase::executor::spawn({

let cancellation_token = cancellation_token.child_token();

async move {

let program_id = mpl_bubblegum::programs::MPL_BUBBLEGUM_ID;

tokio::select! {

_ = cancellation_token.cancelled() => {}

_ = async move {

loop {

match signature_fetcher.fetch_signatures(program_id, args.rpc_retry_interval_millis).await {

Ok(_) => {

metrics_clone.inc_run_fetch_signatures("fetch_signatures", MetricStatus::SUCCESS);

info!("signatures sync finished successfully for program_id: {}", program_id);

}

Err(e) => {

metrics_clone.inc_run_fetch_signatures("fetch_signatures", MetricStatus::FAILURE);

error!("signatures sync failed: {:?} for program_id: {}", e, program_id);

}

}

tokio::time::sleep(Duration::from_secs(60)).await;

}

} => {}

}

}

});

Is not it simpler?

not really, and at this point we give up all the formatting to tokio::select, which is unformattable

kstepanovdev · 2025-02-28T12:12:35Z

nft_ingester/src/bin/migrator/main.rs

+            let cancellation_token = cancellation_token.child_token();
+            async move {
+                loop {
+                    if cancellation_token.is_cancelled() {


The body will be blocked from the cancellation

will be reviewed in the next iterations of this PR

kstepanovdev · 2025-02-28T12:13:34Z

nft_ingester/src/bin/migrator/main.rs

        });

        assets_iter.seek_to_first();
        while assets_iter.valid() {
-            if !rx.is_empty() {
+            if cancellation_token.is_cancelled() {


The body will block the cancellation Maybe it's intended?

will be reviewed in the next iterations of this PR

kstepanovdev · 2025-02-28T12:19:56Z

nft_ingester/src/bin/synchronizer/main.rs

-                    }
-                    Err(e) => {
-                        tracing::error!("Full {:?} synchronization failed: {:?}", asset_type, e);
+        sync_tasks.spawn({


a lot of cancellation_token is happening here. can't it be replaced by one select! ?

kstepanovdev · 2025-02-28T12:21:22Z

nft_ingester/src/cleaners/fork_cleaner.rs


        let mut forked_slots = 0;
        let mut delete_items = Vec::new();

        // from this column data will be dropped by slot
        // if we have any update from forked slot we have to delete it
        for cl_item in self.cl_items_manager.cl_items_iter() {
-            if !rx.is_empty() {
+            if cancellation_token.is_cancelled() {


The huge function is prevented from shutting down by that cancellation because it is happening only at the function beginning.

will be reviewed in the next iterations of this PR

kstepanovdev · 2025-02-28T12:24:05Z

nft_ingester/src/index_synchronizer.rs

    ) {
-        while rx.is_empty() {
-            if let Err(e) = self.synchronize_nft_asset_indexes(rx, run_full_sync_threshold).await {
+        while !cancellation_token.is_cancelled() {


looks like synchronize_nft_asset_indexes already accepts the token, I do not see why we need while here. select may be more suitable here, or even just loop with break on error return from the function

each part in the subsequent call uses loops. the outer usage of cancellation token means that no new iterations will be started on this level, while the cancellation token in child functions acts to break inner loops

kstepanovdev · 2025-02-28T12:24:27Z

nft_ingester/src/index_synchronizer.rs

-        while rx.is_empty() {
-            if let Err(e) =
-                self.synchronize_fungible_asset_indexes(rx, run_full_sync_threshold).await
+        while !cancellation_token.is_cancelled() {


The same concern as above

each part in the subsequent call uses loops. the outer usage of cancellation token means that no new iterations will be started on this level, while the cancellation token in child functions acts to break inner loops

kstepanovdev · 2025-02-28T12:26:02Z

nft_ingester/src/json_worker.rs

-                        let response = json_downloader
-                            .download_file(task.metadata_url.clone(), CLIENT_TIMEOUT)
-                            .await;
+                        let response = tokio::select! {


maybe, that select with loop can replace the while?

will be reviewed in the next iterations of this PR

kstepanovdev · 2025-02-28T12:29:40Z

nft_ingester/src/processors/account_based/mpl_core_fee_indexing_processor.rs

-                    continue;
-                }
+        usecase::executor::spawn(async move {
+            loop {


I'm not sure why we need loop and sleep here simultaneously

nft_ingester/src/init.rs

andrii-kl · 2025-02-28T12:28:47Z

nft_ingester/src/scheduler.rs

+    ) {
+        usecase::executor::spawn(async move {
+            tokio::select! {
+                _ = cancellation_token.cancelled() => {}


Should we terminate all background processes? (as part of a new task)

they will be terminated by select!

kstepanovdev · 2025-02-28T12:36:30Z

nft_ingester/src/processors/accounts_processor.rs

@@ -172,7 +165,7 @@ impl<T: UnprocessedAccountsGetter> AccountsProcessor<T> {
        let mut interval = tokio::time::interval(FLUSH_INTERVAL);
        let mut batch_fill_instant = Instant::now();

-        while rx.is_empty() {
+        while !cancellation_token.is_cancelled() {


while can be replaced with loop inside of the first branch imo

kstepanovdev · 2025-02-28T12:38:50Z

nft_ingester/src/processors/transaction_processor.rs

-    let run_transaction_processor = async move {
-        while rx.is_empty() {
+    usecase::executor::spawn(async move {
+        while !cancellation_token.is_cancelled() {


Cancellation is blocked

kstepanovdev · 2025-02-28T12:40:05Z

nft_ingester/src/rocks_db.rs

    cloned_rocks_storage: Arc<Storage>,
    first_processed_slot_clone: Arc<AtomicU64>,
    last_saved_slot: u64,
 ) -> Result<(), JoinError> {
-    while cloned_rx.is_empty() {
+    while !cancellation_token.is_cancelled() {


while Result(Ok) seems more suitable here

nope, don't agree, it's pretty small and descriptive IMO

kstepanovdev · 2025-02-28T12:42:17Z

rocks-db/src/clients/dump_client.rs

@@ -337,7 +338,7 @@ impl Dumper for Storage {
                    synchronizer_metrics.inc_num_of_records_written("authority", 1);
                }
            }
-            if !rx.is_empty() {
+            if cancellation_token.is_cancelled() {


the cancellation is barely reachable inside this huge function

not really, it will cancel upon subsequent iterator accesses, which are pretty fast

kstepanovdev · 2025-02-28T12:43:09Z

rocks-db/src/fork_cleaner.rs

        let start_time = chrono::Utc::now();
        let mut all_keys = HashSet::new();
        for (key, _) in self
            .db
            .full_iterator_cf(&self.db.cf_handle(RawBlock::NAME).unwrap(), IteratorMode::Start)
            .filter_map(Result::ok)
        {
-            if !rx.is_empty() {
+            if cancellation_token.is_cancelled() {


this should run in parallel imo

it's not async so IMO not worth it

StanChe · 2025-02-28T15:38:45Z

usecase/src/executor.rs

+pub fn spawn<T, F: Future<Output = T> + Send + 'static>(future: F) -> AbortHandle {
+    tokio::task::block_in_place(|| {
+        let mut executor_guard = EXECUTOR.blocking_lock();
+        let _ = executor_guard.get_or_init(JoinSet::new);
+        let executor =
+            executor_guard.get_mut().expect("executor join set to be initialized upon access");
+        executor.spawn(async move {
+            let _ = future.await;
+        })
+    })
+}


block_in_place doesn't seem to be used right here. It temporarily converts the current asynchronous task into a blocking one, ifaik
wdyt of this instead?

Suggested change

pub fn spawn<T, F: Future<Output = T> + Send + 'static>(future: F) -> AbortHandle {

tokio::task::block_in_place(|| {

let mut executor_guard = EXECUTOR.blocking_lock();

let _ = executor_guard.get_or_init(JoinSet::new);

let executor =

executor_guard.get_mut().expect("executor join set to be initialized upon access");

executor.spawn(async move {

let _ = future.await;

})

})

}

pub async fn spawn<T, F: Future<Output = T> + Send + 'static>(future: F) -> AbortHandle {

let mut executor_guard = EXECUTOR.lock().await;

let executor = executor_guard.get_or_init(JoinSet::new);

executor.spawn(async move {

let _ = future.await;

})

}

won't work as i'd like to preserve the signature of spawn being a synchronous function

to use blocking_lock we must enclose the spawning in block_in_place or spawn_blocking or block_on. IMO this is not an issue since this spawn is extremely fast, and we use a multi-threaded runtime anyway, which implies the existence of a blocking thread.

StanChe · 2025-02-28T16:22:18Z

nft_ingester/src/processors/transaction_processor.rs

-                                let current_timestamp = Utc::now().timestamp_millis() as u64;
+                    for tx in txs {
+                        match geyser_bubblegum_updates_processor
+                            .process_transaction(tx.tx, false)


run_until_cancelled assumes the future is safe to cancel, but if you're canceled in the middle of process_transaction - we'll have an issue here

i believe you're right. rolling it back to use the loop

StanChe · 2025-02-28T17:15:01Z

nft_ingester/src/api/backfilling_state_consistency.rs

+                    //     Ordering::Relaxed,
+                    // );
+                    //
+                    tokio::time::sleep(Duration::from_secs(CATCH_UP_SEQUENCES_TIMEOUT_SEC)).await;


why not keep the tokyo select and just replace rx.recv() with cancellation_token.cancelled() in such cases?

good catch, though i will replace with CancellationToken::run_until_cancelled

StanChe · 2025-02-28T17:23:26Z

nft_ingester/src/bin/backfill/main.rs

+    let stop_handle = tokio::task::spawn({
+        let cancellation_token = cancellation_token.clone();
+        async move {
+            usecase::graceful_stop::graceful_shutdown(cancellation_token).await;


I believe it should be ok to set up those interruption handlers twice - here and below, but just double checking

I mean we setup the interruption signal handler twice - in this method and directly in Ctrl+C handler slightly below. That should be ok, just was doublechecking

yeah i think it's fine

nft_ingester/src/bin/backfill/main.rs

Overhaul task spawning in all modules. Remove usage of the `mutexed_tasks` concept, remove usage of a broadcast channel that was used for shutdown, create a static executor to save running tasks & exit gracefully.

Spawn graceful shutdown as the first tokio task in binaries that perform it, await it at the end.

StanChe

Lets' roll this out!

armyhaylenko requested review from StanChe, kstepanovdev and n00m4d February 27, 2025 16:10

armyhaylenko force-pushed the feature/MTG-1216-refactor-shutdown branch 3 times, most recently from 2be95e2 to 72ab5e1 Compare February 28, 2025 10:51

kstepanovdev reviewed Feb 28, 2025

View reviewed changes

andrii-kl reviewed Feb 28, 2025

View reviewed changes

kstepanovdev reviewed Feb 28, 2025

View reviewed changes

andrii-kl reviewed Feb 28, 2025

View reviewed changes

kstepanovdev reviewed Feb 28, 2025

View reviewed changes

kstepanovdev approved these changes Feb 28, 2025

View reviewed changes

armyhaylenko force-pushed the feature/MTG-1216-refactor-shutdown branch from 70f40a6 to 1d5d1d8 Compare February 28, 2025 14:45

andrii-kl approved these changes Feb 28, 2025

View reviewed changes

StanChe reviewed Feb 28, 2025

View reviewed changes

armyhaylenko force-pushed the feature/MTG-1216-refactor-shutdown branch from 1d5d1d8 to 5b58926 Compare March 3, 2025 13:46

armyhaylenko added 4 commits March 3, 2025 16:29

feat: overhaul task spawning in the whole project

f2185df

Overhaul task spawning in all modules. Remove usage of the `mutexed_tasks` concept, remove usage of a broadcast channel that was used for shutdown, create a static executor to save running tasks & exit gracefully.

feat(nft_ingester): spawn graceful shutdown as the first task in main

d2296f9

Spawn graceful shutdown as the first tokio task in binaries that perform it, await it at the end.

feat(ingester,synchronizer): introduce more cancellation checks

fe4c543

chore: remove patch versions from the ecosystem crates

4d7f22d

armyhaylenko force-pushed the feature/MTG-1216-refactor-shutdown branch from 5b58926 to 0c9da15 Compare March 3, 2025 14:29

feat(ingester): replace some loops with run_until_cancelled

f19370b

armyhaylenko force-pushed the feature/MTG-1216-refactor-shutdown branch from 0c9da15 to f19370b Compare March 3, 2025 14:34

StanChe approved these changes Mar 3, 2025

View reviewed changes

armyhaylenko merged commit 427cc6d into develop Mar 3, 2025
9 of 10 checks passed

armyhaylenko deleted the feature/MTG-1216-refactor-shutdown branch March 3, 2025 15:36

github-actions bot mentioned this pull request Mar 10, 2025

Release v0.6.0 #447

Open

feat: overhaul task spawning in the whole project #433

feat: overhaul task spawning in the whole project #433

Conversation

armyhaylenko commented Feb 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrii-kl commented Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kstepanovdev Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kstepanovdev Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armyhaylenko Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrii-kl commented Feb 28, 2025 •

edited

Loading

kstepanovdev Feb 28, 2025 •

edited

Loading

kstepanovdev Feb 28, 2025 •

edited

Loading

armyhaylenko Feb 28, 2025 •

edited

Loading