How to send images to multimodal LLMs safely

Vision-capable language models accept text and images in the same user message. That simple request shape enables document extraction, screenshot debugging, visual search, accessibility, and multimodal support agents.

The production challenge is deciding how images enter the system, what the model is allowed to infer, and how the result is verified.

Use a mixed-content message

Most compatible APIs represent the user content as an ordered array. Each item has a type, such as text or image URL:

const response = await client.chat.completions.create({
  model: process.env.VISION_MODEL,
  messages: [{
    role: "user",
    content: [
      { type: "text", text: "Extract the invoice total and currency." },
      { type: "image_url", image_url: { url: signedImageUrl } },
    ],
  }],
});

Check the selected model’s declared input modalities before sending the call. A text-only endpoint may reject the request or ignore the image.

URL or base64?

Use a short-lived signed URL when the provider can fetch the asset and your security policy permits it. This keeps request bodies small. Use a data URL or base64 payload when the image cannot be hosted, but enforce strict size limits because payload and memory usage grow quickly.

Never expose a permanently public customer file merely to make an API call. Signed URLs should expire quickly, be scoped to one object, and avoid leaking identifying information in the path.

Prepare images deliberately

Higher resolution can improve small-text recognition while increasing latency and token cost. Resize enormous images to the detail needed for the task, retain the original for audit when permitted, and avoid repeated lossy compression.

For multi-page documents, decide whether to send every page, retrieve only relevant pages, or run OCR before the model. Sending everything is simple but often expensive and can distract the model from the evidence that matters.

Request structured output for extraction

If downstream code needs fields, do not parse a narrative answer. Ask for a schema and validate it:

type InvoiceResult = {
  total: number | null;
  currency: string | null;
  evidence: string;
  confidence: number;
};

Treat confidence as a model claim, not proof. Validate currencies, totals, date formats, and required identifiers with deterministic code. Route uncertain or high-value cases to review.

Protect sensitive visual data

Images can contain faces, addresses, account numbers, health information, or screen content outside the user’s intended crop. Before sending them:

1. Confirm the user has authority to process the image.
2. Minimize or redact irrelevant sensitive regions.
3. Apply retention and regional-processing requirements.
4. Record which provider received the asset.
5. Avoid logging raw base64 or signed URLs.

The trace should retain safe metadata and controlled references rather than duplicating the image across every log system.

Evaluate vision workflows with real examples

Build a test set from the image conditions users actually submit: blur, glare, rotation, handwriting, screenshots, dense layouts, multiple languages, and partial crops. Score field accuracy and task completion, not merely whether the answer sounds plausible.

Track results by model, resolution, document type, and preprocessing path. Open the source image and model trace when a score changes.

Observe multimodal agents with Currai

Currai connects image inputs, model responses, tool calls, structured outputs, validation errors, and user corrections. Teams can find recurring visual failure modes and turn them into regression cases without storing unsafe data in ordinary logs.

Get started with the Currai integration skill.

Sources and further reading

• OpenRouter: How to send an image to an LLM
• OpenRouter image-understanding documentation
• Currai: sampling and redacting PII