Multimodal Input

Models with the right capabilities can read images, video and audio alongside your text prompt in /v1/chat/completions — describing a photo, understanding a video clip, or transcribing/reasoning over an audio clip, all in the same request format.

Message format

Instead of a plain string, a message's content becomes an array mixing a text block with one or more media blocks. Three content types are supported, one per media kind — this is the same convention used by OpenAI/OpenRouter, so existing multimodal code ports over directly:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's happening in this clip?" },
    { "type": "image_url", "image_url": { "url": "https://example.com/photo.jpg" } },
    { "type": "video_url", "video_url": { "url": "data:video/mp4;base64,AAAA..." } },
    { "type": "input_audio", "input_audio": { "data": "UklGRi...=", "format": "wav" } }
  ]
}

You don't need to send all three at once — a message can mix just text + image_url, just text + input_audio, etc. Each block is independent and only needs to be included if the model supports it and you actually have that media to send.

Images (image_url)

Two ways to point at an image:

• Public URL — { "type": "image_url", "image_url": { "url": "https://example.com/photo.jpg" } }.
• Base64 data URL — base64-encode the file and send it as data:<mime>;base64,<data> in the same image_url.url field (also accepts raw base64 without the data: prefix).

import base64

with open("photo.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode("utf-8")

completion = client.chat.completions.create(
    model="your-model-id",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
        ],
    }],
)

import fs from "fs";

const b64 = fs.readFileSync("photo.jpg").toString("base64");

const completion = await client.chat.completions.create({
  model: "your-model-id",
  messages: [{
    role: "user",
    content: [
      { type: "text", text: "What's in this image?" },
      { type: "image_url", image_url: { url: `data:image/jpeg;base64,${b64}` } },
    ],
  }],
});

Video (video_url)

Same shape as images, using the video_url type and either a public URL or a base64 data: URL:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Summarize what happens in this video." },
    { "type": "video_url", "video_url": { "url": "data:video/mp4;base64,AAAAIGZ0eXBpc29t..." } }
  ]
}

Only models with capabilities.vision.video: true accept this block — check the model catalog for which ones support video input. Up to 3 videos are accepted per request.

Audio (input_audio)

Audio uses a different shape than image/video — instead of a url field, it's an object with the raw base64 data and a format:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Transcribe and summarize this audio." },
    { "type": "input_audio", "input_audio": { "data": "UklGRiSAAABXQVZFZm10...", "format": "wav" } }
  ]
}

This is for audio the model listens to as input (understanding/transcription) — it's independent from asking a model to speak back, which is a separate opt-in covered below.

Requesting audio back (output modality)

Some chat models can also generate spoken audio as part of their reply, on top of accepting input_audio. This is opt-in per request via top-level modalities and audio fields — it does not require any content block:

{
  "model": "openai/gpt-audio-mini",
  "messages": [{ "role": "user", "content": "Say hello in a friendly tone." }],
  "modalities": ["text", "audio"],
  "audio": { "voice": "alloy", "format": "wav" }
}

If modalities includes "audio" but the chosen model has no audioOutput pricing in the catalog, the request is rejected with a 400 before it's sent upstream. The returned message includes an audio object (id, transcript, base64 data) alongside the usual content, whether or not you set stream — if you request audio without streaming, the server still streams internally and reassembles a single non-streaming response for you.

Limits per request

Media is validated before the request is forwarded to the underlying provider. Anything over these limits returns a 400 immediately, without spending a call upstream:

Size is measured on the base64 payload itself (whether sent as a full data: URL or raw base64). These caps are separate from — and applied before — the model's context-length check on the text portion of your messages.

Checking a model's support

GET /v1/models returns, for every chat model, a capabilities.vision object with the four flags described above. Filter on it client-side before choosing a model for a multimodal request:

{
  "id": "your-model-id",
  "capabilities": {
    "contextLength": 1000000,
    "vision": { "image": true, "video": true, "document": false, "audio": false },
    "thinking": true
  }
}