What PX-bench Is
PX-bench is a long-horizon benchmark from Chordio that evaluates the design decisions coding agents make when adding features to existing products. Rather than judging code in isolation, it measures the product experience an agent produces inside a held-out host app it has never seen.
How It Works
Agents are given a brief and must realize it within a multi-screen host app that carries its own conventions. Each app presents product situations a senior designer would recognize — an implied screen that doesn't exist, state that could be lost on navigation, an ambiguous primary action — and these are mapped in advance so the agent's choices can be scored against a known-good outcome. The harness is Inspect AI, the UK AI Safety Institute's framework, so published scenarios can be independently rerun.
The Eight Scoring Categories
- Intent fidelity — did it build what was asked, working on the happy path?
- Product fit — does the feature attach in the right place and shape?
- Visual craft — is the surface composed into a clear visual hierarchy?
- Convention adherence — does it reuse components, tokens, and house naming?
- Pathway completeness — are all paths and states present and reachable?
- Content & language — are labels, errors, and empty copy right?
- Resilience — does it hold under long content, small screens, and failure?
- Accessibility — keyboard, contrast, labels, and axe violations.
Why It Matters
Teams building coding agents can only improve what they can measure, and product experience is hard to measure. Judging it well usually needs human reviewers, which is expensive, slow, and doesn't keep up with model updates. PX-bench makes product judgment measurable through automatic scoring anchored in expert-defined ground truth, quasi-objective rubrics scoped to where senior designers agree, and failure modes with a known answer.
Running an Eval
You can send your coding agent and harness against the same held-out host apps that scored GPT-5.5, and get back all eight category scores: where product experience holds up, where it breaks, and what it costs to ship.
