Large language models (LLMs) are increasingly capable of generating web pages, but text-driven methods require complex prompts and provide limited control over layout and cross-page visual coherence. Image-driven approaches better reflect real workflows by using UI screenshots, yet existing benchmarks focus primarily on visual fidelity and evaluate interaction only when the target behavior is specified through text or demonstrations. We present UI2App, the first benchmark for evaluating interaction inference: recovering application behavior from screenshots without textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web application. We introduce an end-to-end evaluation pipeline covering executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) measures functional correctness and state-management complexity while rewarding any valid implementation. Experiments on six frontier vision-language models reveal a major gap between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2×. Cross-page state handling remains a major bottleneck. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.
Live recordings of the models' actual generated apps. Two apps, three models each, sorted into three tiers: fully wired, partially wired, and frozen.
/cart with the item + header badge. Fully wired.1 but the cart page still reads
"empty". The handler-to-store wiring is severed (dead code)./cart stays empty.Check has no handler, so it never judges the answer.
Recovered from static screenshots alone: an interactive map, detail drawers, modal forms, data-swapping charts, even a Web-Audio synth.
Automated filtering cuts 2,013 candidate GitHub repositories to 164, and a three-level expert review yields 327 screenshots in 45 state-coherent sets. We select for evidence over scale: every set captures one runnable, multi-route application with enough visual signal to infer its interactions, and the collection spans four functional families and many design systems so scores reflect capability, not memorized templates.
Portfolio · Blog · Documentation · SaaS landing
shadcn / Ant / MUI / Tailwind / Mantine admin panels · CRM · food-ops back-office
E-commerce · Crypto exchange · NFT marketplace · Booking
Analytics dashboard · Music player · Todo · Learning · Intranet
Every generated app is scored on four dimensions.
Does it run? A production build must compile and the home route must render real content. @1 is one-shot; @3 allows up to three rounds of error-feedback repair.
Can you get there? The fraction of screenshot routes a user can actually reach by clicking the app's own navigation.
Does it look right? Judge-free block matching scores Size, Text, Position, Color of the rendered content against the screenshot.
Does the behavior work? A rubric credits any valid implementation of each interaction the screenshots imply, weighted by state complexity (S1 local → S2 shared → S3 cross-route).
Six frontier VLMs, scored end-to-end on the four metrics above. Click any column to sort, switch app category with the tabs, or expand the VFS sub-metrics; rows are ranked by IIS by default.
| # | Model |
|---|
Click any metric header to sort. The best per column is shaded. EXEC = build-and-run · NRS = navigation reachability · VFS = visual fidelity · IIS = interaction inference. All 0–100, higher is better.
Prior benchmarks either drop interaction entirely, or specify the target interaction (via text or demonstration videos). UI2App is the first to require interaction to be inferred from image-only, multi-page input.
| Benchmark | Input | Output | Image-only input |
Multi-page app |
Interaction eval |
Inferred interaction |
|---|---|---|---|---|---|---|
| WebGen-Bench | Text | Framework | ✗ | ✓ | ✓ | ✗ |
| Design2Code | Single image | HTML | ✓ | ✗ | ✗ | — |
| Interaction2Code | Paired images | HTML | ✓ | ✗ | ✓ | ✗ |
| MRWeb | Image + Text | HTML | ✗ | ✓ | ✗ | — |
| Vision2Web (L2) | Images + Text | Framework | ✗ | ✓ | ✓ | ✗ |
| IWR-Bench | Video | HTML | ✗ | ✗ | ✓ | ✗ |
| UI2App (ours) | Images | Framework | ✓ | ✓ | ✓ | ✓ |
@article{ui2app2026,
title = {UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation},
author = {Grace Man Chen and Litao Guo and Yifan Wu and Yiyu Chen and Yenchi Tseng and Sicheng Liu and Yuyu Luo and Ying-Cong Chen},
journal = {arXiv preprint arXiv:2607.06306},
year = {2026}
}