feat(google-common): Grounding with Google Search and Vertex AI Search #7280
Conversation
dosubot
bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
dosubot
bot
added
the
auto:improvement
| tools.forEach((tool) => { | ||
| if ( | ||
| "functionDeclarations" in tool && | ||
| Array.isArray(tool.functionDeclarations) | ||
| ) { | ||
| const funcs: GeminiFunctionDeclaration[] = tool.functionDeclarations; | ||
| geminiTools[0].functionDeclarations?.push(...funcs); | ||
| geminiTools.push({ functionDeclarations: [...funcs] }); |
There was a problem hiding this comment.
I vaguely remember this not working due to Google rejecting it - can you add a test using multiple traditional tools? And maybe one using search grounding alongside several tools as well
There was a problem hiding this comment.
Hello @afirstenberg and @jacoblee93,
Thank you for the review! We’re glad to hear that you appreciate our code contribution. We’ll make sure to add the additional test cases as requested. Please note that libs/langchain-google-vertexai/src/tests/chat_models.int.test.ts already includes test cases for grounding.
There was a problem hiding this comment.
I think there are some known bugs on Google's side about having multiple tools at once - but Google considers this a bug on their end. It should be possible to submit multiple tools at once and mixing functions and other tools.
There was a problem hiding this comment.
Yes this breaks multiple functions, will revert partially
|
CC @afirstenberg for a peek as well! |
jacoblee93
added
question
| @@ -309,6 +309,8 @@ export interface GeminiContent { | |||
|
|
|||
| export interface GeminiTool { | |||
| functionDeclarations?: GeminiFunctionDeclaration[]; | |||
| googleSearchRetrieval?: object; | |||
| retrieval?: object; | |||
There was a problem hiding this comment.
I would have like to see these as actual types or classes rather than as object, but I don't think this should hold off on merging.
|
Generally looks good and I'm sorry I missed this two weeks ago. I do suggest making actual types for the possible attributes for LGTM |
…texAIRetrieval for grounding
jacoblee93
changed the title
jacoblee93
removed
question
|
It's not supported yet from the look of it |
We’ve implemented grounding with Google Search and Vertex AI Search. To use this feature, simply call
.bindTools([searchRetrievalTool])on the model and pass in the appropriatesearchRetrievalToolobject, as defined in the Google Cloud documentation on grounding, depending on the type of grounding you want to use.Here are the implementation details:
googleSearchRetrievalandretrievalparameters toGeminiTool.convertToGeminiToolsandformatToolspreviously supported only one tool type (functionDeclarations), but we’ve refactored them to handle these two new tools.structuredToolsToGeminiToolsfunction has been removed; its logic is now consolidated withinformatTools.We've also added test cases and documentation changes.
Twitter handle: @renathossain
Fixes #5073