Live now · Get your API key

Local visual analysis

Ambient Desktop routes screenshots and images to a MiniCPM-V vision model on your own machine, so text models without native vision can still see, no cloud round trip.

Ambient Desktop's local vision bridge lets a text model understand screenshots, images, and video frames without sending pixels to a cloud service.

Ambient Desktop settings and workspace.
Visual analysis can route to a local MiniCPM-V runtime even when the selected text model lacks native vision.
LOCAL VISION BRIDGEMiniCPM-V · pixels stay on device1Capturevisual inputScreenshot / image / framee.g. proof a UI rendered2Analyze locallyon your machineMiniCPM-V on llama.cppNo cloud endpoint3Returnto the text modelStructured tool resultModel reasons on it4Repairif it failsMissing assets / timeoutDiagnostics guide a fix

A four-stage chain for a look-at-this step in the local vision bridge, using MiniCPM-V so pixels stay on device. Stage one, Capture: a screenshot, image, or video frame enters as the visual input, for example proof a UI rendered. Stage two, Analyze locally on your machine: MiniCPM-V runs on a local llama.cpp runtime with no cloud endpoint. Stage three, Return to the text model: a structured tool result the model can reason about. Stage four, Repair if it fails: missing assets or a timeout are flagged by diagnostics that guide a fix.

When a task needs to understand pixels, the image is routed to a MiniCPM-V model on your own machine and returns a structured result the text model can act on, no cloud round trip.

Vision without leaving your machine#

A coding agent needs to look at things (a screenshot of a failing UI, a diagram, a rendered page), but many strong text models have no native vision, and sending images to a cloud vision service is not always acceptable. Ambient Desktop bridges the gap locally: when a task needs to understand pixels, it routes the image to a MiniCPM-V model running on your own machine and returns structured observations the text model can act on.

How a look-at-this step works#

  1. Capture#

    An image, video frame, or screenshot enters as the visual input; for example, proof that a calculator actually rendered.

  2. Analyze locally#

    MiniCPM-V runs on a local llama.cpp runtime and produces a structured description; the image never goes to a cloud endpoint.

  3. Return to the agent#

    The structured result flows back to the text model as a tool result, so it can reason about what it saw and take the next step.

  4. Repair if needed#

    If the runtime is missing assets, times out, or returns invalid output, diagnostics flag it and guide a repair instead of failing silently.

Platform support#

The local vision runtime ships with platform-appropriate acceleration. Heavier-accelerated platforms get it on by default; lighter ones are experimental.

Platform Support
macOS (Apple Silicon) Metal-accelerated; the local vision runtime is enabled by default
Linux (x64) Vulkan/CUDA acceleration; enabled by default on capable machines
Windows (x64) Experimental and CPU-only today, so analysis is slower; treat it as a preview path

Safety and repair#

  • Local runtime checks surface missing binaries, absent model assets, timeouts, invalid structured output, and memory pressure, with guidance to fix each.
  • Hosted or remote visual endpoints are a separate, higher-risk path that requires explicit security review before they are enabled.
  • Local visual analysis is a routing building block: it lets local-first operation include vision without forcing every screenshot to the cloud.

Keep going#