Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC spec update: Add support for fetching temporal range of recordings #9010

Merged
merged 3 commits into from
Feb 13, 2025

Conversation

zehiko
Copy link
Contributor

@zehiko zehiko commented Feb 12, 2025

What

Adding support for fetching chunks that belong to specific range of the recording. This consists of the 2 APIs calls:

  • fetching chunk ids for a given range
  • fetching chunks for the given chunk ids (which we might want to leverage separately in the future)

As there is a limitation for the client side gRPC streaming on the web (see here), we've decided to send a batch of chunk ids to the server. To make the APIs somewhat symmetrical, the 1st API call will stream batches of chunk ids.

Closes

@zehiko zehiko added exclude from changelog PRs with this won't show up in CHANGELOG.md remote-store remote store gRPC API labels Feb 12, 2025
@zehiko zehiko self-assigned this Feb 12, 2025
Copy link

github-actions bot commented Feb 12, 2025

Web viewer built successfully. If applicable, you should also test it:

  • I have tested the web viewer
Result Commit Link Manifest
f5537c1 https://rerun.io/viewer/pr/9010 +nightly +main

Note: This comment is updated whenever you push a commit.

Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@@ -12,6 +12,9 @@ service StorageNode {
rpc CreateIndex(CreateIndexRequest) returns (CreateIndexResponse) {}
rpc ReIndex(ReIndexRequest) returns (ReIndexResponse) {}

rpc GetChunkIds(GetChunkIdsRequest) returns (stream GetChunkIdsResponse) {}
rpc GetChunks(GetChunksRequest) returns (stream rerun.common.v0.RerunChunk) {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is typically not advised, and I've personally made the mistake of re-using return types before I learned of this advice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a topic here, this already came up. Right now we re-use DataframePart and RerunChunk across more than a few calls, which was intentional and done as part of https://github.com/rerun-io/dataplatform/issues/74.
For now I'd keep it, but let's discuss this re-use and get rid or keep all of them, if that makes sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding pagination support for GetChunks might be a case in point. A typical GetChunksResponse message would include a RerunChunk, and also possibly a breadcrumb to the next page, or even stats.

Temporary consistency for now sounds good though.

@zehiko zehiko merged commit 1402ab9 into main Feb 13, 2025
35 checks passed
@zehiko zehiko deleted the zehiko/temporal-poc branch February 13, 2025 10:26
jprochazk pushed a commit that referenced this pull request Feb 13, 2025
…gs (#9010)

### What

Adding support for fetching chunks that belong to specific range of the
recording. This consists of the 2 APIs calls:
* fetching chunk ids for a given range
* fetching chunks for the given chunk ids (which we might want to
leverage separately in the future)

As there is a limitation for the client side gRPC streaming on the web
(see
[here](#8877 (comment))),
we've decided to send a batch of chunk ids to the server. To make the
APIs somewhat symmetrical, the 1st API call will stream batches of chunk
ids.

Closes
* #8877
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
exclude from changelog PRs with this won't show up in CHANGELOG.md remote-store remote store gRPC API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants