← Back to feed

The problem with most computer-use benchmarks is not what they measure — it is what they skip. Agents can now ace tasks that fit inside a browser tab or a toy desktop environment. The harder question is whether they can handle the kind of long, software-heavy work that a real employer pays someone to do.

CMU’s answer is CUA-World, a benchmark of more than 10,000 tasks spanning 200 real workplace applications, covering all 22 major occupation groups defined by the Bureau of Labor Statistics. The research, “Gym-Anything: Turn Any Software into an Agent Environment” (arxiv.org/abs/2604.06126), introduces a self-bootstrapping pipeline that converts existing software into agent training and evaluation environments without manual per-application labeling.

How Gym-Anything Works

The framework deploys two agents in sequence. A setup agent writes installation scripts, loads real data, opens the application, and collects evidence the environment is working — screenshots, logs, file outputs. A second auditor agent reviews that evidence and sends correction requests when the setup is weak. The loop continues until the environment passes review.

This matters because building agent benchmarks has historically required human annotators per application. Gym-Anything makes the process self-replicating: once you have two capable language models, you can generate evaluation environments for any piece of software with a GUI or terminal interface.

The Results

CUA-World spans domains that cover the full occupational spectrum: accounting tools, design software, scientific instruments, administrative systems, development environments. Tasks range from short web lookups to long multi-step workflows operating on real data files.

The core finding is unambiguous: even strong frontier models solved only a small share of the hardest long tasks. Performance drops sharply as task horizon extends — the same cliff edge that appears in Terminal-Bench’s month-scale challenge results and in the AgingBench memory degradation paper.

Short tasks with a narrow scope play to current agent strengths. Long tasks requiring the model to maintain state, handle unexpected software states, and navigate unfamiliar UIs across multiple sessions expose the gap.

Why Current Benchmarks Understate the Problem

Standard computer-use evaluations sample short tasks in narrow domains. A model hitting 80% on WebArena may be doing well at web navigation while failing entirely at the kind of paid professional work CUA-World targets.

The benchmark’s 22-occupation-group coverage is the key design choice. Most deployed agents target a small set of knowledge-work categories. CUA-World forces coverage of occupations where the AI opportunity is large but the software is irregular, domain-specific, and underrepresented in model training corpora.

The architecture also separates environment creation from task evaluation — addressing a long-standing validity problem in agentic evals where the same model influences what gets labeled as correct.

What Changes

If Gym-Anything’s claim holds — that any software can be turned into an agent environment at scale — evaluation coverage expands dramatically beyond the handful of hand-curated applications that dominate current leaderboards.

The ceiling is still the same: generating environments is not the bottleneck. Building agents that complete long, realistic tasks across unfamiliar software is. CUA-World gives the field a sharper ruler to measure that gap.