How to support image and text input for ChatOpenAI component?

[huu3301]

Hi, I want to use image and text input for the ChatOpenAI component.
And I find that the HumanMessage component has disappeared in the lastest version,
how can I assemble components to form a flow for this requirement by using the latest langflow?
Thank you.

[Amir-m-mohseni]

Hi, I did wrote a small code as a custome component which can recsive a jpg and turns it into a pdf document file from you PC but I have some problems with the embedding and puting in into a FAISS vector store.
here is the code for the custome component:

#JPG loader
from langflow import CustomComponent
from langflow.field_typing import Embeddings, Document
from typing import List
from PIL import Image
import img2pdf
import io
import base64

class JPGLoaderComponent(CustomComponent):
display_name = "JPG Loader"
description = "Load a JPG file, embed it using Azure OpenAI, and convert it to a PDF."
beta = True

def build_config(self):
    return {
        "file_path": {
            "display_name": "File Path",
            "required": True,
            "field_type": "file",
            "file_types": ["jpg", "jpeg"],
            "suffixes": [".jpg", ".jpeg"],
        },
        "output_pdf_name": {
            "display_name": "Output PDF Name",
            "required": True,
            "field_type": "str",
            "value": "output.pdf",
        },
        "embedding": {
            "display_name": "Embedding Model",
            "required": True,
        },
        "code": {"show": False},
    }

def build(self, file_path: str, output_pdf_name: str, embedding: Embeddings) -> List[Document]:
    try:
        # Load JPG image
        with open(file_path, "rb") as image_file:
            image = Image.open(image_file)
            img_byte_arr = io.BytesIO()
            image.save(img_byte_arr, format="JPEG")
            img_byte_arr = img_byte_arr.getvalue()

        # Convert image to base64 string for embedding
        img_base64 = base64.b64encode(img_byte_arr).decode("utf-8")
        print(f"Base64 Image Length: {len(img_base64)}")  # Debug: Print the length of the base64 string

        # Embed the base64 string using the provided embedding model
        embedded_image = embedding.embed_documents([img_base64])
        print(f"Embedded Image Length: {len(embedded_image)}")  # Debug: Print the length of the embedding result

        # Check if embedding was successful
        if not embedded_image or len(embedded_image[0]) == 0:
            raise ValueError("Embedding failed or returned empty result.")
        print(f"Embedding: {embedded_image[0]}")  # Debug: Print the embedding itself

        # Convert image to PDF using img2pdf
        pdf_content = img2pdf.convert(img_byte_arr)

        # Create a Langchain Document with the PDF content and the embedded metadata
        document = Document(
            page_content=pdf_content,
            metadata={"embedding": embedded_image[0]}  # Assuming the embedding returns a list
        )

        print(f"Document Created: {document}")  # Debug: Print the created document
        return [document]
    except Exception as e:
        raise ValueError(f"Error processing JPG file: {e}")

=======================================================================================

[Cristhianzl]

Hi! 👋

We are using other channels as our official means of communication with users. We apologize for the delayed response.
For faster assistance, please direct your questions to our Discord channel or open a GitHub issue, as we are much more active on these platforms.

Langflow Discord Server

Thank you for your understanding.

Best regards,
The Langflow Team