Open benchmark · round 1 · 2026-10-18

Proof, not promises.

Softproof runs each AI design skill on the same briefs as the same model with no skill at all, checks every render with code, has a blind jury score it, and prices it. You see what the skill actually adds.

See the leaderboardBrowse single-shot renders
No skill render for brief card-06
No skillfails safe zones
taste-skill render for brief card-06
taste-skillpasses every check

Same brief (card-06), same model. Over all six briefs the blind jury scored taste-skill 7.76 and the baseline 7.22.

Illustrative pair selected where the baseline and selected condition differ on technical acceptance. The full gallery retains every single-shot output. The Claude juror rates Claude-generated outputs, blinding and equal weighting do not remove family bias, and no human validation has been performed. Scores combine fit to brief, visual quality and brand fit.

Jury's picktaste-skill7.76 / 10 from three blind jurors
Most checks passeddesigner8 of 12 renders
Cheapest accepted renderdesigner$0.0026457 each
No skill at allbaseline4 of 12 renders, jury 7.22

On these six briefs our instruction pack passed 8/12 at $0.0026457 per accepted asset, the best technical pass rate and lowest generation accounting cost per acceptance in the single-shot track. It also had the lowest composite jury mean, 7.17/10, versus baseline 7.22 and taste-skill 7.76. This does not establish general visual or full-workflow superiority.

Leaderboard

Same model. Same briefs. One skill each.

Single shot: the skill's instructions, one model call, no tools, no revisions, on Claude Haiku 5.5. The common configured output setting was 10,000 tokens. One ranked call reported 16,552 aggregate output tokens and one loop call 15,063, mostly thinking tokens. This setting is not a verified hard aggregate cap. The shaded row is the model with no skill. A render counts as accepted only when it passes every code check, and the jury's scores are kept separate from that. Six briefs with one generation each, rendered at two correlated sizes, is a small sample, so read it as what happened on these briefs, not a general ranking. Blind jury: claude-sonnet-5-5, gpt-6.1-sol, gemini-4-argon-medium. The Claude juror rates Claude-generated outputs. Blinding and equal weighting do not remove family bias. No human validation has been performed. The composite averages fit to brief, visual quality and brand fit, rather than visual quality alone.

On these six briefs our instruction pack passed 8/12 at $0.0026457 per accepted asset, the best technical pass rate and lowest generation accounting cost per acceptance in the single-shot track. It also had the lowest composite jury mean, 7.17/10, versus baseline 7.22 and taste-skill 7.76. This does not establish general visual or full-workflow superiority.

#SkillPasses every checkBlind juryCost per accepted assetTime per briefBy check
1 taste-skilljury's pick 50.0%
6/12 assets
7.76 $0.0039985 30.1s
2 ui-ux-pro-max 33.3%
4/12 assets
7.51 $0.0057029 29.2s
3 frontend-design 41.7%
5/12 assets
7.44 $0.0058729 41.8s
4 naksha-studio 58.3%
7/12 assets
7.24 $0.0034401 26.6s
5 baseline
no skill loaded
33.3%
4/12 assets
7.22 $0.0051436 30.9s
6 impeccable 25.0%
3/12 assets
7.19 $0.0077446 31.8s
7 designermost checks passedcheapest accepted render 66.7%
8/12 assets
7.17 $0.0026457 28.5s

Bars, left to right: contrast, overflow, safe zones, type scale, palette, copy. Cost is generation-only CLI list-price accounting on existing subscriptions, including failed outputs, divided by the renders that passed. Jury costs are excluded. These values are not actual billing. Time per brief is generation model-call time, excluding installation and rendering. Full numbers are in summary.json, with receipts and raw juror answers in the results archive.

Loop track

Our designer, with its loop.

Our designer also runs with its render, check and revise loop on the same briefs. That gets extra rounds the single-shot table doesn't allow, so it isn't ranked against the others and has no jury score. Skills built around their own loops get an agentic track next round.

RunPasses every checkGenerations per briefCost per accepted assetTime per brief
designer-loop100.0%
12/12 assets
1.3$0.002584541.3s
Method

How a round works.

Fixed briefs

Copy, audience, sizes and a DESIGN.md brand file. Copy is fixed, so writing isn't judged.

Pinned skills

Installed from their own repos at a commit, never copied here. Same model and configured setting within the single-shot track.

Rendered

Headless Chrome at every size. Scores come from pixels and the DOM, not the model's own account.

Checked

Contrast, overflow, benchmark-policy safe zones, type scale, palette and copy, all in plain code. Safe zones do not guarantee platform crops.

Judged, priced

Three blind jurors from different model families, plus tokens, time and cost per accepted render.

Jurors see shuffled, unlabelled renders and the brief, never skill names or check results. Results get published whoever comes out ahead, including our own designer.

Pinned commits, model IDs and briefs let you rerun the procedure. Outputs are stochastic, the CLI has no supported seed, and identical outputs are not guaranteed. The harness limits matter: filters, blend modes, occlusion, complex masks and path-based lettering need visual review. Raster images, canvas and iframes fail pending separate inspection. Technical acceptance does not establish visual quality.

Add a skill

Built a design skill? Put it on the board.

Open a pull request that adds it to runner/skills.json with its repo, a commit and the path to the skill. The next round runs it. If it needs tools or its own render loop, say so and it goes into the agentic track.

github.com/adityareddy-dev/softproof

git clone https://github.com/adityareddy-dev/softproof
cd softproof && npm ci
npm test
node runner/round.mjs  # Claude Code CLI signed in
node site/build.mjs    # this page, in site/dist