- SketchAgent draws stroke-by-stroke using a learned language, not sketch datasets.
- The AI's strokes proved essential when tested in human collaboration.
- Claude 3.5 Sonnet outperformed GPT-4o as the system's backbone model.
Yael Vinker had a problem that most AI researchers overlook.
The MIT CSAIL postdoc wasn't trying to make AI generate prettier images. She wanted an AI sketch generator which sketches the way a person does: tentatively, stroke by stroke, each mark building on the last. That process, she observed, is how humans externalize half-formed ideas.
When words run out, we reach for a pencil.
The system Vinker and her colleagues built, called SketchAgent, doesn't learn from a library of human drawings. Instead, they gave it a language: a numbered sequence of strokes on a grid, with each stroke labelled by what it represents.
In one training example, the seventh stroke is a rectangle identified as a "front door." The model learns this grammar, then applies it to concepts it has never drawn before: butterflies, DNA helices, the Sydney Opera House.
The AI sketch generator helped researchers make this sketches. Credit: Alex Shipps/MIT CSAIL, with AI-generated sketches from the researchers.
The approach sidesteps a persistent bottleneck in AI art research.
Prior systems that mimicked human sketching were trained on curated datasets of hand-drawn images, which tend to be narrow in scope and uneven in quality. SketchAgent uses an off-the-shelf pretrained model instead, drawing on its broad knowledge to generalize to almost any concept.
What is a multimodal language model?
A multimodal language model trains on both text and images, allowing it to process visual concepts alongside words. SketchAgent uses this visual-linguistic knowledge to plan and sequence drawing strokes, treating a sketch as a kind of sentence.
Key figure
3
language models tested – Claude 3.5 Sonnet produced the most recognizable sketches
Testing Whether the AI Actually Contributes
When Vinker's team tested the system in collaboration mode -- where a human and the model take turns adding strokes to a shared canvas -- they wanted to know whether SketchAgent was genuinely contributing or just along for the ride.
The answer was revealed by subtraction.
In one collaborative sketch of a sailboat, removing the AI-generated strokes left the image unrecognizable. The mast, which SketchAgent had drawn, was the structural element holding the composition together. Similar results appeared across multiple sketch types.
Not everyone is aware of how much they draw in their daily life. We may draw our thoughts or workshop ideas with sketches. Our tool aims to emulate that process, making multimodal language models more useful in helping us visually express ideas.
Yael Vinker, MIT CSAIL postdoc and lead author
The team also compared how different language models performed as the underlying engine. The version built on Claude 3.5 Sonnet produced the most recognizable sketches, outperforming alternatives built on GPT-4o and Claude 3 Opus. Co-author Tamar Rott Shaham, also a CSAIL postdoc, suggested that Claude 3.5 Sonnet appeared to process visual information differently from other models, though she stopped short of offering a mechanism.
Where the AI Sketch Generator Fails
SketchAgent is not a professional drafting tool.
It produces stick figures and simplified doodles, and it struggles with logos, complex creatures, and specific human faces. In collaborative mode, the model occasionally misread what a human partner was drawing. In one case it added a second head to a bunny, misinterpreting which part of its drawing plan the human had addressed.
Vinker attributed the error to the model's use of chain-of-thought reasoning, which breaks tasks into steps that can fall out of sync with a collaborator's actual contributions. The team presented the work at the 2025 Conference on Computer Vision and Pattern Recognition, and see synthetic training data from diffusion models as a path toward sharper handling of complex shapes.
A Different Kind of Interface
The deeper question the research raises is not about sketch quality.
Related reading
The One Thing AI Image Generators Can't Do
They generate endlessly but can't judge what's interesting. New research shows why your taste still matters.
→Text has dominated human-AI interaction since the beginning, partly by default and partly because language models were built around language. SketchAgent suggests that the boundary is more permeable than it appeared.
If a model can be taught a grammar for drawing, the same principle might extend to other visual forms: diagrams, annotations, spatial reasoning on a shared canvas. This can help improve MIT's AI sketch generator moving forward.
Vinker, who received her PhD from Tel Aviv University before joining Professor Antonio Torralba's lab at MIT, has continued presenting the work through late 2025. The sketch, it appears, was never merely decorative.
Sources
- Primary Source: Teaching AI models the broad strokes to sketch more like humans do (MIT News)
- Additional Context:
- SketchAgent: Language-Driven Sequential Sketch Generation (arXiv / CVPR 2025)
- SketchAgent project site (MIT CSAIL)
Fact Check: Claim-by-Claim Verification Verified
All claims verified against MIT News, arXiv paper, and CVPR proceedings. Researcher roles, model comparisons, and quotes confirmed.
Commentary
- SketchAgent produces simplified doodles, not professional-grade drawings.
- The system was also tested with GPT-4o-mini (not mentioned in article).
Sources used for verification
Academic/Peer-reviewed:
Other reliable sources:
Fact-checked by Perplexity Sonar Pro on 2026-03-15
