Large language models (LLMs) have demonstrated growing competence in generating web pages from UI screenshots, which convey both visual structure and cues to application behavior. Yet most screenshot-to-code benchmarks emphasize visual fidelity, while interactive generation benchmarks often supply behavioral specifications or demonstrated transitions. Whether models can infer and realize interactions from static screenshots alone remains insufficiently evaluated. We introduce UI2App to evaluate interaction inference: inferring and realizing application behavior from static visual cues without added behavioral guidance. UI2App comprises 600 screenshots organized into 95 state-coherent sets for runnable multi-route web applications. Our end-to-end pipeline evaluates each artifact along three dimensions: executability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses functional correctness and state-management complexity, crediting valid implementations rather than requiring a match to a single reference. Experiments on six frontier vision-language models reveal a marked mismatch between visual fidelity and interaction realization: the visual-fidelity leader scores only 8.1 on IIS, ranking fourth, while the IIS leader achieves 4.5× that score. High-complexity interactions such as cross-route state persistence remain a major bottleneck, with five of the six models scoring at most 3.5 on this dimension. Overall, these results highlight interaction inference as a key challenge in generating functional web applications from static screenshots.
Live recordings of the models' actual generated apps. Two apps, three models each, sorted into three tiers for the demonstrated behaviors: fully wired, partially wired, and frozen.
/cart with the item + header badge. Fully wired.1 but the cart page still reads
"empty". The handler-to-store wiring is severed (dead code)./cart in this example.Check has no handler, so it never judges the answer.
Recovered from static screenshots alone: an interactive map, detail drawers, modal forms, data-swapping charts, even a Web-Audio synth.
Four-stage automated filtering checks licensing, structure, buildability, and authentication walls, reducing 4,254 candidate GitHub repositories to 423 for expert review. Experts select 95 applications for page-level discriminability, application-level complexity, and corpus-level diversity. The dataset contains 600 screenshots in state-coherent sets, spanning four application categories. Each task provides 3–14 screenshots without action traces or interaction descriptions.
Personal Sites · Publishing & Docs
Business Management · Analytics & Monitoring · General Administration
Marketing Sites · Online Stores · Markets & Booking
Integrated Workspaces · Task-focused Apps
Every generated app is scored on three dimensions.
Does it run? A production build must compile and the home route must render real content. @1 is one-shot; @3 allows up to three rounds of error-feedback repair.
Does it look right? Judge-free block matching scores Size, Text, Position, Color of the rendered content against the screenshot.
Does the behavior work? A rubric scores screenshot-implied interactions by functional correctness and state-management complexity, allowing valid alternative implementations.
Six frontier VLMs on 95 applications, with one generation per application–model pair. IIS is evaluated by GPT-6 Luna (medium) from generated source code and rendered screenshots. Click any column to sort, switch app category with the tabs, or expand the VFS sub-metrics; rows are ranked by IIS by default.
| # | Model |
|---|
Click any metric header to sort. The best per column is shaded. EXEC = build-and-run · VFS = visual fidelity · IIS = interaction inference. All 0–100, higher is better; EXEC is a percentage. Apps failing EXEC@3 receive zero VFS and IIS. Evaluation screenshots are collected by loading routes directly via URL.
Visual-reconstruction benchmarks focus on appearance, while interactive-generation benchmarks often provide behavioral guidance through text, paired states, or demonstrations. UI2App evaluates inferring and realizing interactions from static screenshots without added behavioral guidance.
| Benchmark | Input | Dataset size | Output | Multi-page app | No behavior hints | Interaction eval | Interaction eval size |
|---|---|---|---|---|---|---|---|
| Visual reconstruction | |||||||
| Design2Code | Single image | 484 pages | HTML | ✗ | ✓ | ✗ | — |
| DesignBench–Gen | Single image | 430 pages | HTML / Framework | ✗ | ✓ | ✗ | — |
| MRWeb | Image + Text | 500 samples | HTML | ✓ | ✗ | ✗ | — |
| ComUIBench | Images | 150 pages / 40 sites | HTML | ✓ | ✓ | ✗ | — |
| Interactive HTML generation | |||||||
| Interaction2Code | Paired images | 127 pages | HTML | ✗ | ✗ | ✓ | 374 interactions |
| IWR-Bench | Video | 113 tasks | HTML | ✗ | ✗ | ✓ | 1,001 actions |
| Interactive application generation | |||||||
| WebGen-Bench | Text | 101 tasks | Framework | ✓ | ✗ | ✓ | 647 test cases |
| Vision2Web–L2 | Images + Text | 388 images / 66 apps | Framework | ✓ | ✗ | ✓ | 495 test cases |
| UI2App (ours) | Images | 600 images / 95 apps | Framework | ✓ | ✓ | ✓ | 665 evaluation items |
@article{ui2app2026,
title = {UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation},
author = {Grace Man Chen and Litao Guo and Yifan Wu and Yiyu Chen and Yenchi Tseng and Sicheng Liu and Yuyu Luo and Ying-Cong Chen},
journal = {arXiv preprint arXiv:2607.06306},
year = {2026}
}