DeepSeek V4 doesn't bolt vision on. It rewrites each image as 384 tokens that share the model's main text sequence.

In DeepSeek V4, a picture is not a plugin. It is rewritten into the same 4,096-dim hidden space as text, joined to the same long-context attention, routed by the same MoE experts, and billed to the same 384-token budget per Agent observation. The 32-layer ViT (vision transformer) and the Aligner that sits between it and the V4 main backbone are the mechanism that lets vision live inside the Agent's main loop. The September 1, 2026 open release of the weights and reference inference code lets an outsider audit that design choice, instead of inferring it from API behavior. (Leiphone 雷峰网)

The pipeline starts with a 32-layer ViT front end. Each input image is resized, then cut into 14×14 patches, and each patch is mapped to a 1,024-dimensional vector by 16 attention heads. The grid is then position-encoded with 2D RoPE (rotary position embedding), built separately for the image's horizontal and vertical coordinates. That choice is not incidental. Web and GUI Agent tasks depend on spatial relationships: a button here, a header there, a popup overlapping a chart legend. The 2D RoPE is what makes those relations survive into the backbone. (Leiphone 雷峰网)

Between the ViT and V4's main backbone sits the Aligner. It pools 3×3 neighboring visual features into a single token, sending nine 1,024-dim vectors (a 9,216-dim input) through a two-layer projection that maps 9,216 → 4,096 → 4,096. The 4,096-dim output is the same width as the rest of the V4 backbone. The effect is a roughly 9× compression in each spatial dimension, and a sequence of vision tokens that can sit beside text tokens in the same hidden space without a translation step. (Leiphone 雷峰网)

build_image_block interleaves the compressed visual tokens with IMAGE_START, newline, padding, and IMAGE_END markers. The padding constant COMPRESS_PAD_TO = 4 aligns the visual sequence to four-token boundaries, which matches the 4-compress-ratio layers that V4's main backbone already uses to shrink its long context. Rather than a separate prefix that the language model politely attends to, the image is a marked span of tokens in the same sequence, with its own boundaries and its own alignment to the model's compression schedule. (Leiphone 雷峰网)

merge_image_embeddings then injects the ViT-plus-Aligner visual vectors at the image placeholder positions in the text token stream. From that point on, image and text share the same 4,096-dim space, the same long-context attention window, the same MoE routing, and the same downstream Agent reasoning. The image's special tokens live outside the regular vocabulary of size 129,280, which is how the backbone distinguishes them and routes them through modified attention windows and expert selection. (Leiphone 雷峰网)

The bill arrives at the token budget. vision_max_n_token = 384 is the per-image cap after the Aligner, not a ViT patch count. The DeepSeek API enforces the same 384 cap per image, and that cap directly bounds how much a single Agent observation adds to the long context. A web Agent that takes twenty observations in a single task burns roughly 7,680 tokens to vision alone, before any text reasoning. The wire framing of "V4 sees images" hides that cost. The open code makes it legible. (Leiphone 雷峰网)

For an Agent builder, the design has a direct consequence. Every screenshot and rendered DOM snapshot moves the needle on the same context budget the model uses for instructions, prior tool calls, and scratchpad reasoning. The Aligner decides how many tokens each image actually costs, and the 384 cap is what an outside researcher can verify in the open code rather than estimate from the API. (Leiphone 雷峰网)

The release itself is the relevance clock. DeepSeek opened multimodal access via API on August 21, 2026, then published the weights and reference inference code so the pipeline above can be inspected end-to-end. The Leiphone reading walks the released code rather than the API surface, which is what makes the architectural claim checkable instead of conjectured. V4 reportedly improves on ApexBench, Agents' Last Exam, and Chartography, three multimodal-agent benchmarks. The open release answers one question (what V4's vision actually is) and sets up the next one (whether 384 tokens per observation is a budget a long-horizon Agent can live with). (Leiphone 雷峰网)