Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lack of logging in the langchain_aws package #245

Open
cab938 opened this issue Oct 20, 2024 · 2 comments
Open

Lack of logging in the langchain_aws package #245

cab938 opened this issue Oct 20, 2024 · 2 comments
Labels
enhancement New feature or request good first issue Good for newcomers todo

Comments

@cab938
Copy link

cab938 commented Oct 20, 2024

The langchain_aws package, in particular the ChatBedrockConverse class, has no standard python logging in it. This means visibility into issues is pretty dramatically reduced, and one has to either log at the boto3 level or across into AWS cloud resources. It would be useful to add more common python logging to the various classes in the langchain_aws package.

As an example real use case, I want to see the body of any request and responses sent to boto so I can understand how the language model might be interpreting some of the data. I expected I would see them if I did something like:

logging.getLogger('langchain_aws').setLevel(logging.DEBUG)

However, that doesn't do anything because the module doesn't log, and the best I could do was:

logging.getLogger('boto3').setLevel(logging.DEBUG)

This shows me request headers (not body, which is a problem), and responses (including body, yay!)

@3coins
Copy link
Collaborator

3coins commented Oct 21, 2024

@cab938
LangChain provides some built-in utilities to log application data, that doesn't require adding logging statements directly to the integration classes.

  1. LangSmith is the best tool for reviewing/monitor your langchain application. Here is the setup guide.
    image

  2. Use the set_debug flag. Here is an example:

from langchain_core.globals import set_debug
set_debug(True)

llm = ChatBedrockConverse(model="anthropic.claude-3-haiku-20240307-v1:0")
llm.invoke("What is the capital of France?")

# Output
[llm/start] [llm:ChatBedrockConverse] Entering LLM run with input:
{
  "prompts": [
    "Human: What is the capital of France?"
  ]
}
[llm/end] [llm:ChatBedrockConverse] [261ms] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "The capital of France is Paris.",
        "generation_info": null,
        "type": "ChatGeneration",
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain",
            "schema",
            "messages",
            "AIMessage"
          ],
          "kwargs": {
            "content": "The capital of France is Paris.",
            "response_metadata": {
              "ResponseMetadata": {
                "RequestId": "e411108a-0918-4d28-8030-a9b16f7c8ea1",
                "HTTPStatusCode": 200,
                "HTTPHeaders": {
                  "date": "Mon, 21 Oct 2024 19:00:52 GMT",
                  "content-type": "application/json",
                  "content-length": "212",
                  "connection": "keep-alive",
                  "x-amzn-requestid": "e411108a-0918-4d28-8030-a9b16f7c8ea1"
                },
                "RetryAttempts": 0
              },
              "stopReason": "end_turn",
              "metrics": {
                "latencyMs": 220
              }
            },
            "type": "ai",
            "id": "run-15c8eb2f-7b15-44cd-8544-809ca1692036-0",
            "usage_metadata": {
              "input_tokens": 14,
              "output_tokens": 10,
              "total_tokens": 24
            },
            "tool_calls": [],
            "invalid_tool_calls": []
          }
        }
      }
    ]
  ],
  "llm_output": null,
  "run": null,
  "type": "LLMResult"
}
  1. Use callbacks.
    https://python.langchain.com/docs/how_to/callbacks_runtime/

@3coins 3coins added enhancement New feature or request good first issue Good for newcomers labels Oct 23, 2024
@3coins
Copy link
Collaborator

3coins commented Oct 23, 2024

@cab938
Please feel free to submit a PR to add those additional log statements to the chat model classes; ChatBedrockConverse and ChatBedrock are the best places to start. Let me know if I can help with anything to get you started.

@3coins 3coins added the todo label Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers todo
Projects
None yet
Development

No branches or pull requests

2 participants