Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connect shard throughput #5151

Closed
wants to merge 2 commits into from
Closed

Connect shard throughput #5151

wants to merge 2 commits into from

Conversation

fulmicoton
Copy link
Contributor

@fulmicoton fulmicoton commented Jun 24, 2024

Tested on simian with a pathologic workload:

- chico(rate=40mb)
- chico(rate=10kb)*10

On the left, without any per-shard weighting.
On the right, with per-shard weighting.

indexing throughput
image

cpu usage
image

@fulmicoton fulmicoton force-pushed the connect-shard-throughput branch from 2a7d6ea to 038a316 Compare June 24, 2024 08:56
@fulmicoton fulmicoton force-pushed the connect-shard-throughput branch from 038a316 to edd6787 Compare June 24, 2024 08:58
@fulmicoton fulmicoton requested a review from trinity-1686a June 24, 2024 09:01
Copy link

github-actions bot commented Jun 24, 2024

On SSD:

Average search latency is 1.0x that of the reference (lower is better).
Ref run id: 2268, ref commit: 9fddb68
Link

On GCS:

Average search latency is 1.05x that of the reference (lower is better).
Ref run id: 2269, ref commit: 9fddb68
Link

@fulmicoton fulmicoton force-pushed the connect-shard-throughput branch 4 times, most recently from a43c7a7 to 76cc3d9 Compare July 1, 2024 06:14
@fulmicoton fulmicoton requested a review from trinity-1686a July 1, 2024 06:15
Copy link
Contributor

@trinity-1686a trinity-1686a left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some lints are firing, otherwise lgtm

@fulmicoton
Copy link
Contributor Author

Closed and reopened as #5196

@fulmicoton fulmicoton closed this Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants