Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add completion start time timestamp to relevant generators #8728

Merged
merged 8 commits into from
Jan 17, 2025

Conversation

vblagoje
Copy link
Member

@vblagoje vblagoje commented Jan 16, 2025

Why:

Enhances Langfuse integration to track the time-to-first-token (TTFT) in streaming responses. The new entry in chat message metadata completion_start_time enables TTFT tracking in Langfuse. We'll separately update all other relevant generators in https://github.com/deepset-ai/haystack-core-integrations/

What:

  • Added completion_start_time metadata in the Hugging Face API and OpenAI chat and non-chat generators to record the TTFT.
  • Modified test cases to validate the inclusion of the completion_start_time and its format correctness.

How can it be used:

This change allows developers to track the exact start time of streaming completions, aiding in performance diagnostics

How did you test it:

Included updated unit tests for both Hugging Face API and OpenAI chat and non-chat generators to ensure the presence and correct ISO format of completion_start_time metadata. These tests verify that the timestamps are recorded and formatted correctly during streaming responses.

Notes for the reviewer:

Ensure close attention to the new metadata handling and its correctness across different components. Testing has validated the format, but real integration checks are recommended for full assurance.

@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Jan 16, 2025
@coveralls
Copy link
Collaborator

coveralls commented Jan 16, 2025

Pull Request Test Coverage Report for Build 12825406823

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 13 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.009%) to 91.301%

Files with Coverage Reduction New Missed Lines %
components/generators/hugging_face_api.py 2 96.3%
components/generators/chat/hugging_face_api.py 3 96.46%
components/generators/chat/openai.py 4 96.3%
components/generators/openai.py 4 95.65%
Totals Coverage Status
Change from base Build 12794162077: 0.009%
Covered Lines: 8848
Relevant Lines: 9691

💛 - Coveralls

@vblagoje vblagoje marked this pull request as ready for review January 16, 2025 12:50
@vblagoje vblagoje requested review from a team as code owners January 16, 2025 12:50
@vblagoje vblagoje requested review from dfokina and anakin87 and removed request for a team January 16, 2025 12:50
@vblagoje vblagoje changed the title feat: Add completion start time timestamp to relevant chat generators feat: Add completion start time timestamp to relevant generators Jan 16, 2025
@vblagoje
Copy link
Member Author

vblagoje commented Jan 17, 2025

@julian-risch @anakin87 while at this, I suggest we do the same for the popular generators integrations like ollama, bedrock, and cohere. Or perhaps not - LMK.

@vblagoje vblagoje requested a review from anakin87 January 17, 2025 08:41
Copy link
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of introducing metadata that are very specific (for Langfuse in this case), but I don't see alternative solutions and the PR looks good.

On introducing this metadata in other (Chat)Generators, I would only focus on those supported by our Langfuse implementation: https://github.com/deepset-ai/haystack-core-integrations/blob/72989f4461b863d8be6051dbed5fdefdd6d870ed/integrations/langfuse/src/haystack_integrations/tracing/langfuse/tracer.py#L18

@vblagoje vblagoje merged commit 21dd03d into main Jan 17, 2025
18 checks passed
@vblagoje vblagoje deleted the add_completion_start_time branch January 17, 2025 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:tests type:documentation Improvements on the docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Langfuse integration: completionStartedAt is not logged, required for streaming metrics like TTFT and TPM/TPS
3 participants