500 Error when using Gemini as LLM: Fail to execute model for flow_id: gemini_goldfish_v2_vlp_routed

[palladius]

If I use GoogleGemini as LLM (see code in my forked version), the main.rb demo tends to give consistent 500s after a few invokations.

• It seems like either the API is rate limited, or maybe not
• Errors disappear if I patch Gemini Chat with a manual sleep 0.5

Error from API call

  "error": {
    "code": 500,
    "message": "An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting",
    "status": "INTERNAL",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "Fail to execute model for flow_id: gemini_goldfish_v2_vlp_routed\nError: Internal error occurred."
      }
    ]
  }
}

[palladius]

Referring to this code: https://github.com/palladius/ecommerce-ai-assistant-demo-gemini

By patching the gemini chat() method introducing a manual sleep 0.1 This is how the output is for make test-e2e:

ruby test-e2e.rb
Gemini model: gemini-1.5-pro
I, [2024-09-04T17:55:18.657025 #4151906]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::GoogleGemini
💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤
I, [2024-09-04T17:55:25.886580 #4151906]  INFO -- : [Langchain.rb] [CustomerManagement]: [ 👤 ] Looking up Customer record for [email protected]
I, [2024-09-04T17:55:25.887334 #4151906]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::GoogleGemini
💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤
I, [2024-09-04T17:55:30.675580 #4151906]  INFO -- : [Langchain.rb] [InventoryManagement]: [ 📋 ] Looking up Product SKU: Y3048509
I, [2024-09-04T17:55:30.676136 #4151906]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::GoogleGemini
💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤
0|🧑 [user] 💬 Andrei Bondarev ([email protected]) just purchased 5 t-shirts (Y3048509). His address is 667 Madison Avenue, New York, NY 10065
1|🤖 [model] 🛠️ [1/1] 🛠️  {"name"=>"customer_management__find_customer", "args"=>{"email"=>"[email protected]"}}
2|🔢 [func] 🛠️  customer_management__find_customer => {:id=>2, :name=>"Andrei Bondarev Jr", :email=>"[email protected]"}
3|🤖 [model] 🛠️ [1/1] 🛠️  {"name"=>"inventory_management__find_product", "args"=>{"sku"=>"Y3048509"}}
4|🔢 [func] 🛠️  inventory_management__find_product => {:sku=>"Y3048509", :price=>24.99, :quantity=>0}
5|🤖 [modl] 💬 Sorry, t-shirt Y3048509 is currently out of stock. Would you like to order a different item?
I, [2024-09-04T17:55:32.766466 #4151906]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::GoogleGemini
💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤
I, [2024-09-04T17:55:37.193620 #4151906]  INFO -- : [Langchain.rb] [CustomerManagement]: [ 👤 ] Looking up Customer record for [email protected]
I, [2024-09-04T17:55:37.194132 #4151906]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::GoogleGemini
💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤
I, [2024-09-04T17:55:41.321105 #4151906]  INFO -- : [Langchain.rb] [InventoryManagement]: [ 📋 ] Looking up Product SKU: Z0394853
I, [2024-09-04T17:55:41.321630 #4151906]  INFO -- : [Langchain.rb] [Langchain::Assistant]: Sending a call to Langchain::LLM::GoogleGemini
💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤
String written to file: gemini_http_response_20240904_175550.txt
/usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/llm/google_gemini.rb:95:in `chat': #<Net::HTTPInternalServerError:0x00007f7fc57af3e0> (StandardError)
        from /usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/assistants/assistant.rb:306:in `chat_with_llm'
        from /usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/assistants/assistant.rb:238:in `handle_user_or_tool_message'
        from /usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/assistants/assistant.rb:205:in `process_latest_message'
        from /usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/assistants/assistant.rb:187:in `handle_state'
        from /usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/assistants/assistant.rb:105:in `run'
        from /usr/local/google/home/ricc/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/bundler/gems/langchainrb-860deef98cd6/lib/langchain/assistants/assistant.rb:117:in `add_message_and_run'
        from /usr/local/google/home/ricc/git/ecommerce-ai-assistant-demo-gemini/lib/assistant.rb:6:in `msg'
        from test-e2e.rb:102:in `<main>'
make: *** [Makefile:20: run-test-e2e] Error 1

Captured API response:

==> REQUEST ==>#<Net::HTTP::Post:0x00007f7fc56e5018><== RESPONSE BODY <=={
  "error": {
    "code": 500,
    "message": "An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting",
    "status": "INTERNAL",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "Fail to execute model for flow_id: gemini_goldfish_v2_vlp_routed\nError: Internal error occurred."
      }
    ]
  }
}
==> RESPONSE ==>#<Net::HTTPInternalServerError:0x00007f7fc57af3e0>

[palladius]

Patched chat to write to file any erroring API call to Gemini:

    def chat(params = {})
      sleep 0.1
      puts("💤💤💤 Riccardo Monkeypatching sleeping 0.1sec  💤💤💤")

      params[:system] = {parts: [{text: params[:system]}]} if params[:system]
      params[:tools] = {function_declarations: params[:tools]} if params[:tools]

      raise ArgumentError.new("messages argument is required") if Array(params[:messages]).empty?

      parameters = chat_parameters.to_params(params)
      parameters[:generation_config] ||= {}
      parameters[:generation_config][:temperature] ||= parameters[:temperature] if parameters[:temperature]
      parameters.delete(:temperature)
      parameters[:generation_config][:top_p] ||= parameters[:top_p] if parameters[:top_p]
      parameters.delete(:top_p)
      parameters[:generation_config][:top_k] ||= parameters[:top_k] if parameters[:top_k]
      parameters.delete(:top_k)
      parameters[:generation_config][:max_output_tokens] ||= parameters[:max_tokens] if parameters[:max_tokens]
      parameters.delete(:max_tokens)
      parameters[:generation_config][:response_mime_type] ||= parameters[:response_format] if parameters[:response_format]
      parameters.delete(:response_format)
      parameters[:generation_config][:stop_sequences] ||= parameters[:stop] if parameters[:stop]
      parameters.delete(:stop)

      uri = URI("https://generativelanguage.googleapis.com/v1beta/models/#{parameters[:model]}:generateContent?key=#{api_key}")

      request = Net::HTTP::Post.new(uri)
      request.content_type = "application/json"
      request.body = parameters.to_json

      response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https") do |http|
        http.request(request)
      end

      parsed_response = JSON.parse(response.body)

      wrapped_response = Langchain::LLM::GoogleGeminiResponse.new(parsed_response, model: parameters[:model])

      if wrapped_response.chat_completion || Array(wrapped_response.tool_calls).any?
        wrapped_response
      else
        require 'date'
        timestamp = DateTime.now.strftime('%Y%m%d_%H%M%S')
        filename = "gemini_http_response_#{timestamp}.txt"
        File.open(filename, 'w') do |f|
          f.write("==> REQUEST ==>")
          f.write(request)

          f.write("<== RESPONSE BODY <==")
          f.write(response.body)

          f.write("==> RESPONSE ==>") # => use4less Net::HTTPInternalServerError
          f.write(response.to_s) # => use4less Net::HTTPInternalServerError
        end
        puts("String written to file: #{filename}")

        raise StandardError.new(response)
      end
    end

[palladius]

(note I've only changed the first 2 lines and bloated the raise StandardError.new(response) else condition)

[palladius]

Opened an internal bug: b/364607263