How to send an image to a multimodal embedding model ?

[thoddnn]

I'm starting a llama cpp server using the following command:

llama-server -m "/path/model.gguf" --mmproj "mmproj.gguf"

when I send a HTTP request to http://localhost:8080/embedding with the payload

{
    "content": "Hello"
}

It works and returns an embedding vector, but I would like to send an image instead of text. However, I don't know how to do this. Is it even possible with the current version?

Thanks 🙏

[gnusupport]

I also want to know how can I upload images by using external tools to the server.

[easyfab]

Demo (API) from this PR could perhaps help #12898

import json
import base64
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="sk-test", timeout=9999)

# we support both remove image_url and base64 ; example below is for base64 image (read from disk)

# Function to encode the image
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")


# Path to your image
image_path = "../models/bliss.png"

# Getting the Base64 string
base64_image = encode_image(image_path)

response = client.chat.completions.create(
    model="gpt-4o",
    temperature=0.1,
    stream=True,
    messages=[
        {
            "role": "user",
            "content": [
                { "type": "text", "text": "describe what you see in details" },
                {
                    "type": "image_url",
                    "image_url": {
                        # alternatively, you can put the remote image url here, example: "http(s)://....."
                        "url": f"data:image/png;base64,{base64_image}",
                    },
                },
            ],
        }
    ],
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="")

print("\n\n")

[gnusupport]

Thank you, I'm now using this shell script to describe images in my file manager.

#!/bin/bash

# Check if image file argument is provided
if [ $# -eq 0 ]; then
   echo "Usage: $0 <image-file>"
   exit 1
fi

IMAGE_FILE="$1"
API_URL="http://192.168.1.68:8080/v1/chat/completions"
MODEL="llava"
OUTPUT_FILE="${IMAGE_FILE}.txt"  # Adds .txt to the original filename (e.g., image.jpg.txt)

# Check if file exists
if [ ! -f "$IMAGE_FILE" ]; then
   echo "Error: File '$IMAGE_FILE' not found"
   exit 1
fi

# Create temporary payload file
TMP_PAYLOAD=$(mktemp)

# Generate the JSON payload
cat <<EOF > "$TMP_PAYLOAD"
{
 "model": "$MODEL",
 "messages": [
   {
     "role": "user",
     "content": [
       {"type": "text", "text": "Describe this image in detail"},
       {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,$(base64 -w 0 "$IMAGE_FILE")"}}
     ]
   }
 ]
}
EOF

echo "Generating description for $IMAGE_FILE..."

# Make the API request and save response
curl -s -X POST \
    -H "Content-Type: application/json" \
    -d @"$TMP_PAYLOAD" \
    "$API_URL" | jq -r '.choices[0].message.content' > "$OUTPUT_FILE"

# Clean up
rm "$TMP_PAYLOAD"

echo "Description saved to $OUTPUT_FILE"

[TafadzwaD]

I'm starting a llama cpp server using the following command:

llama-server -m "/path/model.gguf" --mmproj "mmproj.gguf"

when I send a HTTP request to http://localhost:8080/embedding with the payload

{
    "content": "Hello"
}

It works and returns an embedding vector, but I would like to send an image instead of text. However, I don't know how to do this. Is it even possible with the current version?

Thanks 🙏

@thoddnn Yes, you can send images to the /embedding endpoint in llama.cpp, but only if you're using a multimodal model like LLaVA and you’ve specified the --mmproj argument, which you’ve already done.

To include an image in the embedding request, you need to:

• Encode the image in base64.
• Reference it in the content field using an image placeholder like [img-21].
• Provide the base64 image data and the same id (21 in this example) under image_data.

{
  "content": "Image: [img-21].\n Optional Caption",
  "image_data": [
    {
      "id": 21,
      "data": "<BASE64_ENCODED_IMAGE_HERE>"
    }
  ]
}