upload image to base64 LCEL integration

[choiwb]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

def multimodal_prompt(img_base64):        
    return [
    SystemMessage(
    content="""You are specifically response in 3 sentences Korean."""
    ),
    HumanMessage(
            content=[
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{img_base64}", 
                    },
                },
                {
                    "type": "text", "text": "Provide information of given image."
                },
            ]
    )
    ]

class doc_review_system_key(BaseModel):
    문서명: str = Field(description="문서명을 추출해줘.")
    조합명: str = Field(description="조합명(단체명)을 추출해줘, 단 추출 시, 띄어쓰기를 반드시 주의해야 해.")
    고유번호: str = Field(description="고유번호를 추출해줘, 숫자 3자리 - 숫자 2자리 - 숫자 5자리 형태야.")
    주소: str = Field(description="주소(소재지)를 추출해줘.")
    대표자성명: str = Field(description="대표자 성명을 추출해줘.")
    생년월일: str = Field(description="생년월일을 추출해줘.")
    조합인감: str = Field(description="조합인감의 유/무를 O/X로 나타내줘.")

json_parser = JsonOutputParser(pydantic_object=doc_review_system_key)

doc_review_system_key_prompt = PromptTemplate(
    template="Answer the user question.\n{format_instructions}\n{question}\n",
    input_variables=["question"],
    partial_variables={"format_instructions": json_parser.get_format_instructions()},
)

aws_vis_pipe = RunnablePassthrough() | sonnet_llm | StrOutputParser()
aws_vis_json_parser_pipe = doc_review_system_key_prompt | sonnet_llm | json_parser

                    img_context_total = {}
                    for i in images_base64:    
                        img_context = aws_vis_pipe.invoke(multimodal_prompt(i)) 
                        img_parser_context = aws_vis_json_parser_pipe.invoke({"question": img_context})

Description

above of code, both LCEL, “aws_vis_pipe”, “aws_vis_json_parser_pipe”.
how to iget to integration of 2 LCEL?????

System Info

.

[dosubot[bot]]

To integrate the two LangChain Execution Layers (LCELs) aws_vis_pipe and aws_vis_json_parser_pipe, you need to ensure that the output from aws_vis_pipe is correctly formatted and compatible with the input requirements of aws_vis_json_parser_pipe. Here’s how you can achieve this:

1. Invoke aws_vis_pipe: This pipeline processes the image and returns a string output.
2. Format the Output: Ensure the output from aws_vis_pipe is structured as a question for the aws_vis_json_parser_pipe.
3. Invoke aws_vis_json_parser_pipe: This pipeline takes the formatted output and processes it further.

Here’s the modified code to integrate the two LCELs:

def multimodal_prompt(img_base64):        
    return [
        SystemMessage(
            content="You are specifically response in 3 sentences Korean."
        ),
        HumanMessage(
            content=[
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{img_base64}", 
                    },
                },
                {
                    "type": "text", "text": "Provide information of given image."
                },
            ]
        )
    ]

class doc_review_system_key(BaseModel):
    문서명: str = Field(description="문서명을 추출해줘.")
    조합명: str = Field(description="조합명(단체명)을 추출해줘, 단 추출 시, 띄어쓰기를 반드시 주의해야 해.")
    고유번호: str = Field(description="고유번호를 추출해줘, 숫자 3자리 - 숫자 2자리 - 숫자 5자리 형태야.")
    주소: str = Field(description="주소(소재지)를 추출해줘.")
    대표자성명: str = Field(description="대표자 성명을 추출해줘.")
    생년월일: str = Field(description="생년월일을 추출해줘.")
    조합인감: str = Field(description="조합인감의 유/무를 O/X로 나타내줘.")

json_parser = JsonOutputParser(pydantic_object=doc_review_system_key)

doc_review_system_key_prompt = PromptTemplate(
    template="Answer the user question.\n{format_instructions}\n{question}\n",
    input_variables=["question"],
    partial_variables={"format_instructions": json_parser.get_format_instructions()},
)

aws_vis_pipe = RunnablePassthrough() | sonnet_llm | StrOutputParser()
aws_vis_json_parser_pipe = doc_review_system_key_prompt | sonnet_llm | json_parser

img_context_total = {}
for i in images_base64:    
    # Invoke the first pipeline
    img_context = aws_vis_pipe.invoke(multimodal_prompt(i))
    
    # Format the output for the second pipeline
    formatted_question = {"question": img_context}
    
    # Invoke the second pipeline
    img_parser_context = aws_vis_json_parser_pipe.invoke(formatted_question)
    
    # Store the result
    img_context_total[i] = img_parser_context

Explanation:

1. multimodal_prompt Function: Generates the prompt for the image in base64 format.
2. doc_review_system_key Class: Defines the schema for the JSON output.
3. json_parser: Parses the output into the defined schema.
4. doc_review_system_key_prompt: Creates a prompt template with format instructions.
5. aws_vis_pipe: Processes the image and returns a string.
6. aws_vis_json_parser_pipe: Takes the formatted string and processes it into the JSON schema.

By ensuring the output from aws_vis_pipe is correctly formatted as a question for aws_vis_json_parser_pipe, you can seamlessly integrate the two LCELs.

To continue talking to Dosu, mention @dosu.