Seeing is prompting: a vision-model tutorial

Paste a screenshot into a chatbot and ask "what's wrong here?" — congratulations, you just wrote a vision prompt. Modern chatbots are vision-language models (VLMs): the image is part of the prompt, and the words you attach to it shape what the model notices, describes, and extracts.

Vision prompting has quietly become one of the most useful AI skills: debugging UIs, reading dashboards, explaining diagrams, transcribing whiteboards. But vision models do not see the way you do. This tutorial maps what they do well, where they fail, and the concrete techniques that get better results.

What a vision model actually sees#

Under the hood, a VLM is two systems glued together: a visual encoder that converts an image into tokens, and a language model that reasons over those tokens. The visual encoder — typically from the CLIP family — processes the image as a grid of patches, compresses thousands of pixels into a few hundred visual tokens, and hands that summary to the language model.

The key implication: the model never sees your original image. It sees a compressed, downsampled interpretation of it. Fine print, small objects, and subtle textures can be lost at this bottleneck before the language model ever gets a vote. Everything below follows from this.

What vision models do well#

Screenshots. This is the sweet spot. Vision models can explain what's on screen, read error dialogs, walk through a confusing settings page, and summarize a dashboard. A strong practical workflow: screenshot a bug, paste it in, and ask for the likely cause and fix. Developers widely use this for screenshot-to-code — describing a UI mockup as HTML and CSS — where current frontier models (Claude, GPT, Gemini) produce plausible first drafts.

Diagrams and charts. Flowcharts, architecture diagrams, network diagrams, org charts: models are generally good at following boxes, arrows, and labels and explaining the logic of a diagram. For charts, they can read legends and axes, describe trends, and answer "what was Q3 revenue?" — as long as the chart is legible at the resolution the model actually receives.

Documents. Receipts, invoices, forms, menus, whiteboards, nutrition labels — modern VLMs handle OCR-like extraction on clear, well-lit documents and beat dedicated OCR pipelines on convenience: no separate toolchain, and the extraction comes with reasoning attached (sums line items, flags inconsistencies). Dense or hand-written text is where this degrades fast.

Product photos and scenes. Describing a photo, comparing two product shots, assessing visible damage, generating alt text for accessibility — all reliable territory. Accessibility apps like Be My Eyes have shipped vision-model-powered scene description to visually impaired users for years, which is a decent signal that everyday scene understanding works.

Where they fail#

Counting and precise spatial reasoning. This is the best-documented blind spot. The arXiv paper "Vision language models are blind" (Rahmanzadehgervi et al., 2024) tested GPT-4o, Gemini 1.5 Pro, and two Claude models on tasks trivial for humans — counting overlapping circles, figuring out how many rings are interlocked — and all four performed poorly unless they'd been trained on nearly identical images. Counting more than a handful of objects, reading analog clocks, and judging exact spatial relationships remain unreliable, even in newer models.

Pixel precision. Asking a model for exact coordinates ("where is the Submit button, in pixels?") is asking for trouble. The language model reasons about a compressed representation, not the pixel grid. For agent and automation work, practitioners separate the jobs: let the vision model describe what to interact with in natural language ("the blue Submit button in the form footer"), and resolve coordinates with a separate method — template matching, an accessibility tree, or numbered on-screen markers (more on this below).

Fine detail in compressed or low-quality images. Low-resolution images, heavy JPEG compression, blur, darkness, and noise all degrade performance sharply. The fix is boring but effective: send the sharpest, highest-resolution crop of the relevant region. A 4K screenshot downscaled by the chat app is often worse than a tight crop of the relevant panel at native resolution.

Hallucinating objects that aren't there. Vision models inherit the captioning style of their training data: human-written captions describe what's salient, not everything present, so models learn what a typical scene mentions. The result is confident descriptions of objects that don't exist. Worse in pipelines: when a VLM can't read a field in a document, its language decoder may fill the gap with a plausible default or silently omit it — returning a structurally valid JSON payload with missing or invented values. Deterministic OCR, by contrast, fails loudly with noise. If you're extracting structured data from documents, treat "it looked fine" as no evidence at all: the failure mode is silent.

Text-rich complexity. Dense tables, multi-page documents, and small rotated or stylized text remain weak points. An OCR benchmark on Arabic documents (Hennara, 2026) found a generalized vision model failing completely on complex layouts while a specialized OCR pipeline succeeded — a reminder that "reads text in images" is not "reads all text in all images."

Techniques that actually help#

Put the image first, then the text. Models process images more reliably when the image precedes the question. When sending several images, label them ("Image 1: dashboard; Image 2: error log") so references stay unambiguous.

Ask targeted questions, not "describe this." "Describe this" gets a generic caption. "List every labeled component in this diagram and the direction of each arrow" gets a structured answer. Specificity focuses the model's attention on the regions that matter.

Crop and zoom. The single highest-leverage trick. If the relevant detail occupies 5% of the screenshot, crop to it before sending. Anthropic's own prompting guidance notes that giving the model a "crop tool" — the ability to zoom into regions of an image — consistently boosts performance on image evaluations, enough that they've published a cookbook for it. You can do the manual version in any image editor.

Set-of-Mark prompting. For grounding tasks (pointing at specific UI elements or regions), overlay numbered markers on the image and ask the model to return a marker number instead of coordinates. This was introduced for GPT-4V by Yang et al. (2023) and has become a de facto standard in visual agents: picking "marker 7" is far more reliable than estimating raw pixel positions. A simpler cousin — a labeled grid (A1, B2, C3…) over the screenshot — works for coarse navigation.

Request confidence, not just answers. When asking a model to read a blurry or partially visible value, add: "If you cannot read it clearly, say so and give your confidence level." This converts silent hallucination into an explicit uncertainty signal you can act on.

Constrain the output. For extraction, specify a schema and an escape hatch: "Return JSON with keys X, Y, Z. If a field is not visible in the image, set it to null — never guess." Forbidding guesses is doing real work here, because the default behavior of the language decoder is to keep the text flowing.

Verify with a second pass. For anything that matters — invoices, measurements, medical or legal documents — ask the model to quote the exact visible text or describe the visual evidence supporting its answer, then check it yourself. Human review isn't optional for high-stakes visual tasks; it's the workflow.

Quick-reference: prompt recipes by input type#

Takeaway#

Treat a vision model like a very smart colleague looking at your screenshot through frosted glass: excellent at gist, structure, and reasoning — unreliable at fine print, counting, and precise location. Crop ruthlessly, ask specific questions, constrain the output format, and verify anything that matters. Images are prompts now; learn to write them well.