Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Small changes: code reuse, simplify, doc comments, client timeouts #25

Merged
merged 9 commits into from
May 9, 2024

Conversation

declark1
Copy link
Collaborator

@declark1 declark1 commented May 9, 2024

This PR is a batch of several small changes:

  • Adds input_masks(), input_detectors(), and output_detectors() helper methods to GuardrailsConfig to reuse code shared by unary and streaming task handlers.
  • Moves detection task spawning code from chunk_and_detect() to detect(), consistent with chunk().
  • Updates detect() and chunk() code to be easier to follow for new Rustacians
    • Tokio task handles are collected to tasks and separately, try_join_all() is used to await them, collecting results to results
  • Added doc and general comments to orchestrator task handlers
  • Dropped unneeded tokio::spawn from handle_chunk_task() and handle_detection_task()
  • Added default connect and request timeouts for http & grpc clients
    • This is to override actual client defaults which are too high, configurable service-level timeouts will be added in #17
  • Added request_id to new requests
  • Updated "handling task" (info level) log event to include request_id and exclude inputs text (also addresses #18)
  • Fixed broken test_deserialize_config() unit test (tls is currently a required field, although it should be Option)

@declark1 declark1 requested a review from gkumbhat May 9, 2024 16:32
@@ -92,7 +94,8 @@ async fn stream_classification_with_gen(
State(state): State<Arc<ServerState>>,
Json(request): Json<models::GuardrailsHttpRequest>,
) -> Result<impl IntoResponse, (StatusCode, Json<String>)> {
let task = StreamingClassificationWithGenTask::new(request);
let request_id = Uuid::new_v4();
Copy link
Collaborator

@gkumbhat gkumbhat May 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally we would be getting a request id / transaction id from our upstream and it would be good to stick those in logs for tracking purpose and use this as a "default", if not provided. Would you expect any issues with replace this request_id later on? Since you have added it already in downstream functions, it would essentially look like replace request_id with something from request (although it may be in header, so we may need to do some processing)

Copy link
Collaborator Author

@declark1 declark1 May 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't instrumented tracing here yet, but I think the trace-id should probably be used for tracking across services? We should probably still have a unique service-level request identifier though, which is what this is for.

If there are additional transaction id(s) provided from upstream services via headers, we can still extract and record them. I haven't seen anything documented on this to know what to expect.

model_id = %task.model_id,
config = ?task.guardrails_config,
"handling task"
);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q - can we still tell from this info log vs. the streaming one that this is the "unary" task vs. the "streaming" task?

Copy link
Collaborator Author

@declark1 declark1 May 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, no. I was thinking to add detail once we implement streaming and/or additional methods.

I could make this simply "handling unary task" and "handling streaming task", but I assumed there could be additional task types added. I was trying to avoid "handling classification with gen task" and "handling streaming classification with gen task" as these are verbose.

Ideally, these API method names (and associated task handlers) can be renamed/simplified (generate and generate_stream would be nice)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll change it to "handling unary task" and "handling streaming task" for now.

Copy link
Collaborator

@evaline-ju evaline-ju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@declark1 declark1 merged commit b6f5030 into main May 9, 2024
1 check passed
@declark1 declark1 deleted the dc-tweaks branch May 9, 2024 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants