🧹 batch sync assets #995
Merged
🧹 batch sync assets #995
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ivan Milchev <[email protected]>
imilchev
force-pushed
the
ivan/batch-sync-assets
branch
from
7be5579 to
fe16ac1
Compare
Signed-off-by: Ivan Milchev <[email protected]>
|
cnquery/cnspec and its providers are in general not thread safe and doing this will trigger data races. Not sure how to get this information for the providers, but you can at least see the data races fairly quickly for cnspec: |
|
it looks like those are related to the |
Signed-off-by: Ivan Milchev <[email protected]>
|
The scans are now synchronous, batch size is 100 |
czunker
approved these changes
Dec 13, 2023
|
I gave this a try with a container registry scan. Works as expected. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of calling
SynchronizeAssetsonce for all discovered assets, we create batches. This makes sure that if we have discovered a huge amount of assets, the call would still succeed.I intentionally put a low number for the batch size because with this implementation, batches are scanned in parallel, which also decreases the total scan time.
Downside of this approach is that the progress bar results ordering is no longer deterministic. However, since this is a progress bar, I don't think ordering should matter