Support field fileData (direct file URL) for Gemini models

[vricciardulli]

This PR addresses the feature request described in issue #1134

The implementation proposed here is straightforward: add a FileUrl class that can be used in user prompts in the same way that DocumentUrl and similar are used.

The main downside I see with this approach is that this is only supported by Gemini models on Vertex AI and, in minor measure, by GLA-Gemini models, while the FileUrl can in principle be used with any model (and would lead to a RuntimeError exceptions being raised in some models).

[hyperlint-ai]

PR Change Summary

Added support for the FileUrl class to enable direct file URL input for Gemini models on Vertex AI, enhancing the input capabilities for users.

• Introduced the FileUrl class for direct file URL input.
• Updated documentation to include instructions for using FileUrl with Gemini models.
• Clarified that only Gemini models on Vertex AI support this feature.

Modified Files

• docs/input.md

---

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

[vricciardulli]

Also allowed FileUrl to be passed to GLA-Gemini models, as it is supported as the docs show. Although that documentation does not provide much information and PDF files don't seem to be supported. YouTube URLs seem to be supported by the GLA.

[ioga]

Thanks a lot for making this PR, it worked for me. Would love to see it get merged. Couple suggestions:

1. Since youtube videos is the only supported type, I think the example should use them instead of the pdf URLs which do not work.
2. media_type field is not required for youtube URLs to work, it should be made optional.

[vricciardulli]

Thanks a lot for making this PR, it worked for me. Would love to see it get merged. Couple suggestions:

1. Since youtube videos is the only supported type, I think the example should use them instead of the pdf URLs which do not work.
2. media_type field is not required for youtube URLs to work, it should be made optional.

Thanks for the interest!

Did you try this with both GLA and Vertex providers? because PDF URLs seem to work with Vertex. I agree that with GLA only youtub URLs seem to work though.

[ioga]

Thanks a lot for making this PR, it worked for me. Would love to see it get merged. Couple suggestions:

1. Since youtube videos is the only supported type, I think the example should use them instead of the pdf URLs which do not work.
2. media_type field is not required for youtube URLs to work, it should be made optional.

Thanks for the interest!

Did you try this with both GLA and Vertex providers? because PDF URLs seem to work with Vertex. I agree that with GLA only youtub URLs seem to work though.

Ah I see, yeah I've only tried GLA. I want this PR for youtube links support specifically anyway.

[vricciardulli]

@Kludex in case you had time to give this a quick look: regarding the comments above, do you think it would be better to add a youtube link in the docs instead of a PDF url?

Also, unsure if this approach is correct: maybe it's better to be able to set this via GeminiModelSettings, as David suggested some time ago?

[ioga]

Can you just add both youtube and PDF examples?

My second point, making media type optional, is probably more material: right now Gemini seems to ignore it and you can just pass whatever, but no way to know if google will change that behavior in the future.

[vricciardulli]

@ioga added, let me know what you think. Still need to increase test coverage and maybe for the Vertex provider the mime type should be inferred when possible, because it is required for that provider.

[ioga]

works for me on google-gla, thank you.