You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Be sure to set GOOGLE_API_KEY in your environment for use.
Features added:
Chat completion with support for message arrays conforming to OpenAI spec
google content array "parts" supports "model" (instead of "assistant") and "user", while system messages are may be included in the systemInstruction field
Support for streaming chat completion
Function calling - streaming and non-streaming
Support for Google AI Cache Manager - cache data may be added through createCacheManager and used by passing in the cache name from the response into chat.completions.create by setting it in { additionalProperties: cacheName }} (note: this requires a paid account)
Note: polyfill added due to missing support for TextEncoderStream in bun.js (oven-sh/bun#5648)
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Cool stuff @jcamera! Have you thought about how this might interact with/support the caching features that Gemini has? It'd be an awesome capability to add.
Cool stuff @jcamera! Have you thought about how this might interact with/support the caching features that Gemini has? It'd be an awesome capability to add.
Hey thank you @ZECTBynmo! I'm just getting my feet wet here, but I did look at an example using GoogleAICacheManager. We could totally add methods on the Google provider to support this, and use the same input data types as the chat completion.
And we could then add a param on chat.completions.create like { cache: name } using the name returned when the cache is set. This would diverge from the OpenAI spec a bit, but shouldn't hurt anything as an extra optional field. If OpenAI eventually adds a cache field we'd need to update. :-)
I can start adding this into this PR unless any objections.
Hi @ZECTBynmo, fyi I added some pieces to allow using the Google cache manager. (A paid account is needed to use it) Added an example in /examples. I'd be curious to learn about your use cases, and happy to help if any issues! 🙂
jcamera
changed the title
[WIP] adding google support for llm-client (aka llm-polyglot)
add google support for llm-client (aka llm-polyglot)
Jul 15, 2024
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
jcamera
changed the title
llm-polyglot support for Google AI
This is using Google AI SDK (aka Gemini API) for Javascript - https://github.com/google-gemini/generative-ai-js
Be sure to set GOOGLE_API_KEY in your environment for use.
Features added:
systemInstructionfieldcreateCacheManagerand used by passing in the cache name from the response intochat.completions.createby setting it in{ additionalProperties: cacheName }}(note: this requires a paid account)Note: polyfill added due to missing support for TextEncoderStream in bun.js (oven-sh/bun#5648)
we may also consider using their Vertex AI API which has some tentative support for using OpenAI libraries with Gemini - https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-gemini-using-openai-library
or directly using their Gemini Rest API - https://ai.google.dev/api/rest
Feedback is welcome :-)