[Bug, Claude models]: "AnthropicError: No content in response" when content is empty array

[hi019]

What happened?

Sometimes Anthropic returns an empty content array. LiteLLM doesn't expect this and throws an error:

AnthropicError: No content in response

During handling of the above exception, another exception occurred:

APIError                                  Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/litellm/utils.py](https://localhost:8080/#) in exception_type(model, original_exception, custom_llm_provider, completion_kwargs, extra_kwargs)
   8092                     else:
   8093                         exception_mapping_worked = True
-> 8094                         raise APIError(
   8095                             status_code=original_exception.status_code,
   8096                             message=f"AnthropicException - {original_exception.message}. Handle with `litellm.APIError`.",

APIError: AnthropicException - No content in response. Handle with `litellm.APIError`.

Repro:

import litellm
litellm.set_verbose=True
litellm.drop_params=True

await litellm.acompletion(
    model='anthropic/claude-3-opus-20240229',
    api_key='sk-ant',
    messages=[
        {
            'role': 'system',
            'content': "You will be given a list of fruits. Use the submitFruit function to submit a fruit. Don't say anything after."
        },
        {
            'role': 'user',
            'content': "I like apples"
        },
        {
            'content': "<thinking>The most relevant tool for this request is the submitFruit function.</thinking>",
            'role': 'assistant',
            'tool_calls': [
                {
                    'function': {
                        'arguments': '{"name": "Apple"}',
                        'name': 'submitFruit'
                    },
                    'id': 'toolu_012ZTYKWD4VqrXGXyE7kEnAK',
                    'type': 'function'
                }
            ]
        },
        {
            'role': 'tool',
            'content': '{"success":true}',
            'tool_call_id': 'toolu_012ZTYKWD4VqrXGXyE7kEnAK'
        }
    ],
    max_tokens=2000,
    temperature=1,
    tools=[
        {
            'type': 'function',
            'function': {
                'name': 'submitFruit',
                'description': 'Submits a fruit',
                'parameters': {
                    'type': 'object',
                    'properties': {
                        'name': {'type': 'string', 'description': 'The name of the fruit'}
                    },
                    'required': ['name']
                }
            }
        }
    ],
    frequency_penalty=0.8
)

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

Hi @hi019 this is not a bug, it's expected behaviour since openai's pydantic object expects content if not none, to be a non-empty string.

What would you expect to happen here?

[krrishdholakia]

closing as this isn't a "bug", but i'm open to your feedback on this @hi019

[paneru-rajan]

I was able to replicate this, here is more on this:

The original_exception.status_code is 200

The original_exception is AnthropicError('No content in response')

Entire Traceback:

status_code: 200
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
python-BaseException
Traceback (most recent call last):
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/main.py", line 320, in acompletion
    response = await init_response
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/llms/anthropic.py", line 306, in acompletion_function
    return self.process_response(
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/llms/anthropic.py", line 143, in process_response
    raise AnthropicError(
litellm.llms.anthropic.AnthropicError: No content in response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers-pro/pydevd_asyncio/pydevd_nest_asyncio.py", line 135, in run
    return loop.run_until_complete(task)
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers-pro/pydevd_asyncio/pydevd_nest_asyncio.py", line 238, in run_until_complete
    return f.result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/another.py", line 10, in test_get_response
    resp = await litellm.acompletion(
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/utils.py", line 3697, in wrapper_async
    raise e
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/utils.py", line 3529, in wrapper_async
    result = await original_function(*args, **kwargs)
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/main.py", line 341, in acompletion
    raise exception_type(
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/utils.py", line 9120, in exception_type
    raise e
  File "/Users/rajan/work/personal/adelaide/sem1/spi/try-lite-llm/.venv/lib/python3.9/site-packages/litellm/utils.py", line 8051, in exception_type
    raise APIError(
litellm.exceptions.APIError: AnthropicException - No content in response. Handle with `litellm.APIError`.

Ran following in the console:
original_exception
AnthropicError('No content in response')

original_exception.status_code
200

Which was thrown from
FIle:litellm/llms/anthropic.py:142
Code:

elif len(completion_response["content"]) == 0:
    raise AnthropicError(
        message="No content in response",
        status_code=response.status_code,
    )

Since the status code from Anthropic is 200, If we want consistency and always expect our library to give a consistent output across the board we can raise a different exception like EmptyContent, which then can be handled by the client differently.

[paneru-rajan]

Or we can do something like: elif response.status_code != 200 and len(completion_response["content"]) == 0: let me know if either of these makes sense.

[krrishdholakia]

@paneru-rajan i haven't tried the initial prompt yet. is that a consistent repro?

If it's inconsistent -> i'm assuming an error makes sense to allow some retry logic to occurr

if it's consistently returning an empty response -> what would you want to happen here? @hi019 @paneru-rajan

[paneru-rajan]

I tried couple of time exact payload @hi019 attached, it is consistently returning an empty response for me so far.

I also think error makes sense but raising generic error AnthropicError, it might be impossible for the client to catch empty response and handle this, as retry would not make sense if the response is consistent.

 raise AnthropicError(
                message="No content in response",
                status_code=response.status_code,
            )

Let's wait for @hi019 for his take on this, then we can take steps or close this for good.

---

I don't recommend this: We can do something to suppress it:

# old code: elif len(completion_response["content"]) == 0:
elif response.status_code != 200 and len(completion_response["content"]) == 0:

---

Recommended approach:
Define a new Exception:

class AnthropicMissingContentError(AnthropicError):
    ...

then raise it:

 raise AnthropicMissingContentError(
                message="No content in response",
                status_code=response.status_code,
            )

[krrishdholakia]

@paneru-rajan we don't raise a generic anthropic error.

We raise an 'APIError' -> we map our exceptions to the openai exceptions. https://platform.openai.com/docs/guides/error-codes/api-errors

This maps to a '500' error which indicates a server-side error, and is an acceptable error code for retrying -

litellm/litellm/utils.py

Line 7629 in fa47ce4

def _should_retry(status_code: int):

In this case - it seems like an error is ok but the error code is not a retry-able exception?

[krrishdholakia]

@paneru-rajan Appreciate the work, but please don't recommend closing discussions which are still ongoing.

If an error is raising a 200 status code, then that's an issue we should be fixing.

[krrishdholakia]

fixed the status code issue (We now return the right code) + added testing - b5f3f19

[paneru-rajan]

Anthropic status 200 with empty content is being replaced by LiteLLM APIError's 500.

500 status code like i said will not be a accurate reflection but given limited list of Exception, i get the point. Thanks for clarification. @krrishdholakia .

[krrishdholakia]

That's a good point - so i'm guessing we shouldn't be retrying on this error, since it is a consistent repro.

A 400 error would make more sense then - since it would be considered an unretriable error. Thoughts?

[hi019]

Thanks for the info everyone. I agree that a non-retryable 400 status makes more sense here than a 500.