Support streaming inputs in CLI tooling #1477
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change does not integrate streaming into report collector yet as we're not quite ready for it. But it provides a means to split a `BufRead` interface into `M` streams that read from it one at a time. This should not require reading the entire file in memory in order to submit it. There are a few things that I decided not to bring in the first version: * Reading from a file can be done in async runtime using `AsyncRead` interface. The benefit of it is not clear to me atm. * reading more than one line at a time. We could support reading lets say 8kb chunks from file and then let HTTP streams to stream it out.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1477 +/- ##
==========================================
- Coverage 93.86% 93.78% -0.08%
==========================================
Files 235 236 +1
Lines 42451 42837 +386
==========================================
+ Hits 39845 40174 +329
- Misses 2606 2663 +57 ☔ View full report in Codecov by Sentry. |
eriktaubeneck
approved these changes
Dec 5, 2024
cberkhoff
reviewed
Dec 5, 2024
| // but probably it does not matter for this code as it is not on the hot path | ||
| if state.idx == self.idx { | ||
| let poll = state.read_next(); | ||
| if poll.is_ready() { |
There was a problem hiding this comment.
does read_next only return Poll::Ready?
There was a problem hiding this comment.
no, but only if it is ready, we want to advance the index
cberkhoff
approved these changes
Dec 5, 2024
|
|
||
| proptest! { | ||
| #[test] | ||
| fn proptest_round_robin(input: Vec<String>, count in 1_usize..953) { |
There was a problem hiding this comment.
what is this secret number?
There was a problem hiding this comment.
nothing really, just something that is not large enough to cause proptests to run forever
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change does not integrate streaming into report collector yet as we're not quite ready for it. But it provides a means to split a
BufReadinterface intoMstreams that read from it one at a time. This should not require reading the entire file in memory in order to submit it.There are a few things that I decided not to bring in the first version:
AsyncReadinterface. The benefit of it is not clear to me atm.