Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recreate PR #2600 and refresh branch #2801

Closed
wants to merge 78 commits into from
Closed

Conversation

kmrmt
Copy link
Contributor

@kmrmt kmrmt commented Jan 15, 2025

Description

Recreate #2600 and refresh branch.

  • update rust dependencies
  • meta.proto and payload.proto has already merged to main
  • move test_client to meta crate

Related Issue

Versions

  • Vald Version: v1.7.15
  • Go Version: v1.23.4
  • Rust Version: v1.83.0
  • Docker Version: v27.4.0
  • Kubernetes Version: v1.32.0
  • Helm Version: v3.16.3
  • NGT Version: v2.3.5
  • Faiss Version: v1.9.0

Checklist

Special notes for your reviewer

Summary by CodeRabbit

  • Dependencies

    • Updated multiple dependencies across various Rust packages to newer versions
    • Added new dependencies for observability and metadata handling
  • Toolchain

    • Upgraded Rust toolchain from version 1.83.0 to 1.84.0
  • New Features

    • Added a test client for metadata service
    • Enhanced observability and request interception functionality
    • Implemented improved error handling and logging for metadata operations
  • Refactoring

    • Streamlined initialization of meter and tracer providers
    • Updated dependency management and version control

@CLAassistant
Copy link

CLAassistant commented Jan 15, 2025

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
8 out of 9 committers have signed the CLA.

✅ kmrmt
✅ vdaas-ci
✅ vankichi
✅ kpango
✅ takuyaymd
✅ highpon
✅ datelier
✅ aknishid
❌ smorihira
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Contributor

coderabbitai bot commented Jan 15, 2025

Important

Review skipped

More than 25% of the files skipped due to max files limit. The review is being skipped to prevent a low-quality review.

74 files out of 260 files are above the max files limit of 75. Please upgrade to Pro plan to get higher limits.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough

Walkthrough

This pull request introduces comprehensive updates across multiple Rust packages, focusing on dependency version upgrades and observability enhancements. The changes span several components, including the meta service, agent, and observability libraries. Key modifications include updating dependencies like opentelemetry, tokio, and tonic, introducing a new test client for the meta service, and improving tracing and logging capabilities. The Rust toolchain itself was also upgraded from version 1.83.0 to 1.84.0.

Changes

File Change Summary
rust/bin/meta/Cargo.toml Added dependencies: kv, opentelemetry, prost-types, sled, defer, observability. Updated tokio and tonic versions. Added new binary target test_client.
rust/bin/meta/src/handler.rs Introduced new Meta struct with store and bucket fields, and a new constructor method.
rust/bin/meta/src/handler/meta.rs Enhanced get, set, and delete methods with improved error handling and logging.
rust/bin/meta/src/main.rs Added observability and request interception functionality. Introduced MetadataMap struct and intercept function.
rust/bin/meta/src/test_client.rs New test client implementation for metadata service with trace context injection.
Multiple Cargo.toml files Version updates for dependencies like anyhow, prost, tokio, tonic, cxx, miette, opentelemetry
rust/rust-toolchain and rust/rust-toolchain.toml Rust toolchain updated from 1.83.0 to 1.84.0

Sequence Diagram

sequenceDiagram
    participant Client as Test Client
    participant MetaService as Metadata Service
    participant Observability as Observability System
    
    Client->>Observability: Inject Trace Context
    Client->>MetaService: Set Key-Value
    MetaService-->>Client: Confirmation
    Client->>MetaService: Get Value
    MetaService-->>Client: Return Value
    Client->>MetaService: Delete Key
    MetaService-->>Client: Deletion Confirmation
Loading

Possibly related PRs

Suggested Labels

type/bug, size/XL, area/internal, area/observability, area/dependency-update

Suggested Reviewers

  • hlts2

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

cloudflare-workers-and-pages bot commented Jan 15, 2025

Deploying vald with  Cloudflare Pages  Cloudflare Pages

Latest commit: 64a8af7
Status: ✅  Deploy successful!
Preview URL: https://bff5f090.vald.pages.dev
Branch Preview URL: https://feature-meta-rebase-main.vald.pages.dev

View logs

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (8)
rust/bin/meta/src/main.rs (2)

28-44: Implement Clone for MetadataMap for Potential Future Use

While the current implementation of MetadataMap works, consider implementing Clone or Copy traits for MetadataMap to enhance its usability, especially if you plan to pass instances across threads or tasks in the future.


46-51: Handle Possible Errors in the Interceptor Function

The intercept function currently does not handle any potential errors that might occur during the extraction of the parent context. Although unlikely, it's good practice to handle or log unexpected situations to aid in debugging.

Consider adding error handling or logging within the intercept function:

 fn intercept(mut req: Request<()>) -> Result<Request<()>, tonic::Status> {
     let parent_cx =
-        global::get_text_map_propagator(|prop| prop.extract(&MetadataMap(req.metadata())));
+        global::get_text_map_propagator(|prop| prop.extract(&MetadataMap(req.metadata()))).clone();
     req.extensions_mut().insert(parent_cx);
     Ok(req)
 }
rust/bin/meta/src/handler/meta.rs (2)

65-91: Simplify Nested match Statements and Improve Error Messages

The set method has deeply nested match statements, which can be simplified for better readability. Additionally, the error messages could be more consistent.

Refactor the code to flatten the match statements:

 let key_value = request.into_inner();

-let key = match key_value.key {
-    Some(k) => k.key,
-    None => {
-        ctx.span().add_event("Invalid argument", vec![KeyValue::new("error", "Key is missing")]);
-        return Err(tonic::Status::invalid_argument("Key is missing"));
-    }
-};

-let value = match key_value.value {
-    Some(v) => match v.value {
-        Some(any_value) => any_value.value,
-        None => {
-            ctx.span().add_event("Invalid argument", vec![KeyValue::new("error", "Value is missing")]);
-            return Err(tonic::Status::invalid_argument("Value is missing"));
-        }
-    },
-    None => {
-        ctx.span().add_event("Invalid argument", vec![KeyValue::new("error", "Value is missing")]);
-        return Err(tonic::Status::invalid_argument("Value is missing"));
-    }
-};
+if key_value.key.is_none() || key_value.value.is_none() {
+    ctx.span().add_event("Invalid argument", vec![KeyValue::new("error", "Key or Value is missing")]);
+    return Err(tonic::Status::invalid_argument("Key or Value is missing"));
+}
+
+let key = key_value.key.unwrap().key;
+let value = if let Some(any_value) = key_value.value.unwrap().value {
+    any_value.value
+} else {
+    ctx.span().add_event("Invalid argument", vec![KeyValue::new("error", "Value is missing")]);
+    return Err(tonic::Status::invalid_argument("Value is missing"));
+};

119-128: Handle Deletion of Non-Existent Keys Gracefully

When deleting a key that does not exist, the method may return an error or succeed silently. Consider defining the expected behavior and handling it accordingly.

Perhaps inform the client if the key did not exist:

 match self.bucket.remove(&raw_key) {
     Ok(Some(_)) => {
         ctx.span().add_event("Key deleted successfully", vec![KeyValue::new("key", key)]);
         Ok(tonic::Response::new(Empty {}))
     },
+    Ok(None) => {
+        ctx.span().add_event("Key not found", vec![KeyValue::new("key", key)]);
+        Err(tonic::Status::not_found("Key not found"))
+    },
     Err(e) => {
         ctx.span().add_event("Failed to delete key", vec![KeyValue::new("error", e.to_string())]);
         Err(tonic::Status::internal(format!("Failed to delete key: {}", e)))
     }
 }
rust/bin/meta/src/handler.rs (1)

29-30: Close the Store Properly on Shutdown

Ensure that the Store is properly closed when the application shuts down to prevent data corruption.

Implement the Drop trait or provide a shutdown method for Meta:

impl Drop for Meta {
    fn drop(&mut self) {
        // Close the store
        if let Err(e) = self.store.close() {
            eprintln!("Failed to close store: {}", e);
        }
    }
}
rust/bin/meta/src/test_client.rs (2)

48-48: Consider configurable endpoint.

The endpoint is hardcoded to "http://[::1]:8081". Consider making it configurable through environment variables or command-line arguments.

+use std::env;
+
 #[tokio::main]
 async fn main() -> Result<(), Box<dyn std::error::Error>> {
-    let mut client = MetaClient::connect("http://[::1]:8081").await?;
+    let endpoint = env::var("META_SERVICE_ENDPOINT")
+        .unwrap_or_else(|_| "http://[::1]:8081".to_string());
+    let mut client = MetaClient::connect(endpoint).await?;

130-134: Improve error handling in get operation.

The error handling for the get operation after deletion only prints the error. Consider implementing proper error handling or recovery logic.

-    match response {
-        Ok(res) => println!("Get key: aaa, RESPONSE={:?}", res.into_inner()),
-        Err(e) => println!("Get key: aaa failed with error: {:?}", e),
-    }
+    match response {
+        Ok(res) => println!("Get key: aaa, RESPONSE={:?}", res.into_inner()),
+        Err(e) => {
+            if e.code() == tonic::Code::NotFound {
+                println!("Key 'aaa' was successfully deleted and is not found");
+            } else {
+                eprintln!("Unexpected error while getting key 'aaa': {:?}", e);
+            }
+        }
+    }
rust/bin/meta/Cargo.toml (1)

32-35: Consider adding integration tests for the test client.

While the test client is marked as doc = false, it would be beneficial to have integration tests to ensure it works correctly with the metadata service.

Consider adding a tests directory with integration tests for the test client functionality.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4c73960 and 379fe66.

📒 Files selected for processing (5)
  • rust/bin/meta/Cargo.toml (1 hunks)
  • rust/bin/meta/src/handler.rs (1 hunks)
  • rust/bin/meta/src/handler/meta.rs (1 hunks)
  • rust/bin/meta/src/main.rs (1 hunks)
  • rust/bin/meta/src/test_client.rs (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: check-format-diff
  • GitHub Check: runner / textlint
  • GitHub Check: Cloudflare Pages
  • GitHub Check: Analyze (go)
🔇 Additional comments (5)
rust/bin/meta/src/main.rs (1)

96-96: Ensure Proper Shutdown of Observability System

The observability.shutdown()? call might return an error that is currently being ignored. To ensure proper resource cleanup, handle any potential errors during shutdown.

Check if the shutdown method can fail and handle the error accordingly:

 observability.shutdown()?;
+println!("Observability system shut down successfully.");

Or handle the error:

-if let Err(e) = observability.shutdown() {
-    eprintln!("Failed to shut down observability system: {}", e);
-}
rust/bin/meta/src/handler/meta.rs (1)

79-91: Validate Any Message Types

When handling prost_types::Any, it's important to validate the type_url to ensure that the received message is of the expected type.

Add validation for the type_url:

if any_value.type_url != "type.googleapis.com/expected.MessageType" {
    ctx.span().add_event("Invalid message type", vec![KeyValue::new("type_url", any_value.type_url)]);
    return Err(tonic::Status::invalid_argument("Invalid message type"));
}
rust/bin/meta/src/handler.rs (2)

28-32: Ensure Thread Safety and Consider Using a Singleton for the Store

The store is wrapped in an Arc, which is good for sharing across threads. Ensure that the Store and Bucket types are thread-safe if they will be accessed concurrently.

Check the documentation for the kv crate to confirm that Store and Bucket are safe to share between threads. If they are not, consider using synchronization primitives or limiting access to a single thread.


30-30: ⚠️ Potential issue

Handle Possible Errors When Creating the Bucket

When creating the bucket with store.bucket, consider handling the possibility that the bucket already exists or cannot be created.

Add error handling or check for existing buckets:

 let bucket = store.bucket::<Raw, Raw>(Some("meta_bucket"))?;
+// Alternatively, check if the bucket exists and handle accordingly

Likely invalid or redundant comment.

rust/bin/meta/Cargo.toml (1)

22-24: Verify dependency versions for security and compatibility.

Several new dependencies have been added. Please verify:

  1. Check for any known vulnerabilities in these versions
  2. Ensure compatibility with the rest of the ecosystem

Also applies to: 26-26

✅ Verification successful

Dependencies are consistently versioned across the codebase

The new dependencies are properly aligned:

  • prost-types 0.13.2 is consistently used across all Rust components
  • kv 0.24.0 and opentelemetry 0.23.0 are cleanly introduced only in the meta component

Note: The API documentation mentions an older version of prost-types and might need updating, but this is not a blocking issue.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check for known vulnerabilities in dependencies
cargo audit
# List all dependency versions for verification
cargo tree

Length of output: 218


Script:

#!/bin/bash
# Find all Cargo.toml files
echo "=== Cargo.toml files ==="
fd Cargo.toml

# Search for kv dependency versions
echo -e "\n=== kv dependency versions ==="
rg "kv\s*=\s*\"[^\"]*\"" -A 1 -B 1

# Search for opentelemetry versions
echo -e "\n=== opentelemetry versions ==="
rg "opentelemetry\s*=\s*\"[^\"]*\"" -A 1 -B 1

# Search for prost-types versions
echo -e "\n=== prost-types versions ==="
rg "prost-types\s*=\s*\"[^\"]*\"" -A 1 -B 1

Length of output: 1594

rust/bin/meta/src/main.rs Show resolved Hide resolved
rust/bin/meta/src/main.rs Outdated Show resolved Hide resolved
rust/bin/meta/src/handler/meta.rs Show resolved Hide resolved
rust/bin/meta/src/handler/meta.rs Show resolved Hide resolved
rust/bin/meta/src/test_client.rs Show resolved Hide resolved
rust/bin/meta/src/test_client.rs Show resolved Hide resolved
@github-actions github-actions bot added size/XL and removed size/L labels Jan 15, 2025
@kmrmt kmrmt changed the title Feature/meta/rebase main Recreate PR #2600 and clean up branch Jan 15, 2025
@kmrmt kmrmt changed the title Recreate PR #2600 and clean up branch Recreate PR #2600 and refresh branch Jan 15, 2025
@kmrmt kmrmt marked this pull request as ready for review January 15, 2025 08:04
@kmrmt kmrmt requested review from a team and kpango and removed request for a team January 15, 2025 08:05
highpon and others added 21 commits January 30, 2025 13:02
* fix go build argument

* update go
This commit fixes the style issues introduced in 8179e7d according to the output
from Gofumpt and Prettier.

Details: None

Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
* fix validation logic for manager indexer

Signed-off-by: aknishid <[email protected]>

* Change validation to include zero.

Signed-off-by: aknishid <[email protected]>

---------

Signed-off-by: aknishid <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
* add Search_Config validation

* change to error function and generate individual errors

* add search configuration
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: Kosuke Morimoto <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: Kosuke Morimoto <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: Kosuke Morimoto <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Signed-off-by: Kosuke Morimoto <[email protected]>
@kmrmt kmrmt dismissed stale reviews from kpango, vankichi, and coderabbitai[bot] via b4ca563 January 30, 2025 04:13
@kmrmt kmrmt force-pushed the feature/meta/rebase-main branch from 9023068 to b4ca563 Compare January 30, 2025 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants