Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multimodal inputs for inference #239

Merged
merged 7 commits into from
Dec 4, 2023

Conversation

isaac-chung
Copy link
Contributor

@isaac-chung isaac-chung commented Dec 4, 2023

Support multimodal inputs for inference without using private methods

from clarifai.client.model import Model
from clarifai.client.input import Inputs

prompt = "What time of day is it?"
image_url = "https://samples.clarifai.com/metro-north.jpg"
inference_params = dict(temperature=0.2, max_tokens=100)

-text_pb = Inputs.get_input_from_bytes("", text_bytes=prompt.encode()).data.text
-image_pb = Inputs.get_input_from_url("", image_url= image_url).data.image

# Model Predict
model_prediction = Model("https://clarifai.com/openai/chat-completion/models/openai-gpt-4-vision").
-    predict(inputs = [Inputs._get_proto(input_id="",imagepb=image_pb,text_pb=text_pb)],
+    predict(inputs = [Inputs.get_multimodal_input(input_id="",image_url=image_url, raw_text=prompt)],
inference_params=inference_params)

print(model_prediction.outputs[0].data.text.raw)

clarifai/client/input.py Outdated Show resolved Hide resolved
@isaac-chung isaac-chung requested a review from sainivedh December 4, 2023 14:36
@isaac-chung isaac-chung marked this pull request as ready for review December 4, 2023 14:37
@isaac-chung isaac-chung requested review from sanjaychelliah and removed request for sainivedh December 4, 2023 14:59
clarifai/client/input.py Outdated Show resolved Hide resolved
clarifai/client/input.py Outdated Show resolved Hide resolved
@isaac-chung isaac-chung requested a review from sainivedh December 4, 2023 15:58
Copy link
Contributor

@sainivedh sainivedh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@isaac-chung isaac-chung merged commit ed8f698 into master Dec 4, 2023
17 checks passed
@isaac-chung isaac-chung deleted the isaac/support-multimodal-inputs-for-inference branch December 4, 2023 16:28
@sainivedh sainivedh mentioned this pull request Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants