Screen-using AI gets a Visual Skill Card for 'where am I in this task?'

A screen-using AI can label every button on a page and still lose track of which step it is on — login, checkout, or done. A new arXiv preprint, SkillLens, packages that missing knowledge as Visual Skill Cards: compact per-workflow records that name the current step, point to the on-screen evidence that would prove it, and define what counts as success.

The two-stage system first turns past browser interactions into these cards, then, at inference, retrieves only the ones a frozen vision-language model needs. On the public Multimodal-Mind2Web benchmark, the authors report the setup lifts a frozen GPT-5.4-mini executor by 11.6 points in step success rate and 2.9 points overall; on WebLINX-BrowserGym the gains hold. A companion training method, CardDistill, distills the same workflow knowledge into a small Qwen3-VL-2B student that runs without card retrieval at inference and posts a similar 12.0 / 3.2 point lift.

The honest read: a frozen executor handed the right evidence is not the same as an agent that learned the workflow end-to-end, and the gains are on two browser benchmarks that may favor this style of memory. The paper does not test redesigned interfaces, novel workflows, or adversarial UIs. What it does show is a clean pattern — step state for GUI tasks can be packaged as data, retrieved when needed, or baked into smaller models.