Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor file source #5100

Closed
wants to merge 5 commits into from
Closed

Refactor file source #5100

wants to merge 5 commits into from

Conversation

rdettai
Copy link
Contributor

@rdettai rdettai commented Jun 10, 2024

Description

To prepare the S3 backed SQS source, this PR refactors the file source so that generates RawDocBatch can be reused.

How was this PR tested?

Added extra unit tests to the new DocFileReader.

@rdettai rdettai self-assigned this Jun 10, 2024
@rdettai rdettai force-pushed the refactor-file-source branch from eb6a8cf to c9b594d Compare June 10, 2024 12:22
@rdettai rdettai changed the base branch from sqs to main June 10, 2024 12:24
@rdettai rdettai force-pushed the refactor-file-source branch from 962decc to 008bfe3 Compare June 10, 2024 12:42
@rdettai rdettai requested a review from guilload June 10, 2024 12:46
@rdettai rdettai force-pushed the refactor-file-source branch from 008bfe3 to 373435a Compare June 10, 2024 12:53
Copy link

github-actions bot commented Jun 10, 2024

On SSD:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 2337, ref commit: 4ade7b5
Link

On GCS:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 2339, ref commit: 4ade7b5
Link

@rdettai rdettai force-pushed the refactor-file-source branch from 373435a to afd7868 Compare June 11, 2024 07:01
@rdettai rdettai force-pushed the refactor-file-source branch 6 times, most recently from fdc98a5 to 3555d3a Compare June 12, 2024 21:06
@rdettai rdettai force-pushed the refactor-file-source branch from 94f94ee to bea5d82 Compare June 26, 2024 07:56
@rdettai rdettai force-pushed the refactor-file-source branch from bea5d82 to 6d6fdb1 Compare June 26, 2024 08:38
@rdettai
Copy link
Contributor Author

rdettai commented Jul 4, 2024

Closing this in favor of #5148 that brings significant changes on top of the file source,

@rdettai rdettai closed this Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants