Parametric CAD Bench V3: Text, Editing, and Engineering Drawings

Today we are introducing Parametric CAD Bench v3, a 100-task benchmark that evaluates three distinct CAD workflows: 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 image-to-CAD tasks based on engineering drawings.

GPT-6 Astra leads the public v3 leaderboard at 56.87%, only 0.12 percentage points ahead of Claude Fable 5.1 at 56.75%. Their 95% confidence intervals overlap substantially, as does Claude Opus 5's third-place interval, so the point-estimate ordering among the top three should not be read as a statistically decisive separation on this 100-task cohort.

Fable records the highest creation score, 55.48%, and image-to-CAD score, 72.71%. Astra leads create-and-edit at 44.37% and reaches 69.70% on image-to-CAD.

The v3 leaderboard

Overall score is the mean continuous reward across all 100 tasks. A perfect result earns 1.0, partial results receive partial credit, and failed or unscored trials count as zero. The 95% confidence interval is calculated over the 100 per-task rewards. Every row links to its public Harbor run. The live leaderboard is available here.

[1] The row is labeled high by effective reasoning effort. Grok Build 1.0.30 ignored the requested xhigh setting because it was unsupported, leaving Grok 4.6 at its default high effort. The linked job is an audited, credential-sanitized mirror. ↩

What changed in v3

The benchmark now covers three CAD workflows. V2 contained 100 text-to-CAD creation tasks. V3 uses a new 30/30/40 cohort spanning creation, create-and-edit, and image-to-CAD. The image tasks provide an engineering drawing rather than a natural-language feature specification, while create-and-edit tests whether an agent can understand and modify an existing parametric model.

The verifier moves to gnucleus-freecad-validator 0.6.0 with the v2 scorer. The runtime remains pinned to FreeCAD 1.1.0, while the updated scorer uses an oriented-bounding-box size gate, stricter per-component comparisons, revised geometry weights, and stronger spatial alignment scoring. The new results therefore measure a different evaluation contract from v2.

V3 closes a reward-hacking avenue exposed by v2. In v2 trajectories, we observed agent-model systems attempting to locate or download the held-back reference CAD or the validator and grader artifacts instead of solving the modeling task from the provided inputs. Whether or not a particular attempt succeeded, seeking the scoring target is outside the intended benchmark contract and can contaminate the measurement. The observation does not imply that every v2 system behaved this way or that every v2 score was affected, but it showed that stronger runtime isolation was necessary.

Execution is more tightly isolated. Held-back reference geometry remains in the separate verifier environment. Agent execution defaults to no network except for the selected model provider, and task instructions restrict work to task-required files and state under /app.

The run contract is frozen and auditable. V3 uses Harbor 0.23.0 and task schema 1.3 against one immutable dataset digest. Each official row is a fresh 100-task cohort with one attempt per task and four-way concurrency. Only narrowly defined infrastructure failures may be replaced, with the original attempt retained; model, tool, timeout, verifier, and output failures otherwise remain zero.

What the first v3 results show

The leading systems separate most clearly on image-to-CAD. Fable reaches 72.71%, Astra 69.70%, and Opus 62.64%. Across all 11 rows, image-to-CAD averages 32.28%, versus 42.54% for creation and 30.74% for create-and-edit, but image-to-CAD also has the largest variation between systems.

Create-and-edit contains persistent hard cases. Six of the 100 tasks receive zero reward from every evaluated row, and all six are create-and-edit tasks. Eleven tasks receive a perfect score from at least one row. This leaves substantial headroom even though individual tasks can be solved consistently.

When the “expensive” model is cheaper

Model tier or per-token price is a poor proxy for the cost of completing an agentic workload. GPT-6 Astra is both the highest-scoring row and the most cost-efficient row in this matrix: it costs $317.81 for the full cohort, or $5.59 per percentage point of leaderboard score. It is cheaper in absolute terms than five of the other ten systems—Grok 4.6, Grok 4.7, Muse Spark 1.3, Claude Fable 5.1, and Claude Opus 5—and costs only $25.82 more than GPT-5.6 Sol while scoring 26.89 percentage points higher.

The likely mechanism is efficiency rather than a paradox in model pricing: a stronger model-agent system can reach a useful solution with fewer turns, shorter contexts, and less recovery work. Astra reports the fewest tokens in the matrix despite leading the benchmark.

The agent interface matters even when the model label is held constant. Gemini 3.8 Flash paired with mini-swe-agent (with multimodal bridge) reaches 36.19% overall and 31.10% on image-to-CAD, compared with 27.56% overall and 17.45% on image-to-CAD for Antigravity. This is not a perfectly controlled agent ablation—the prompts, tool protocols, and serving paths differ—but it shows that the way an agent exposes images can materially change the measured capability of the same named model. The bridge is therefore worth examining in detail: it gives mini-swe-agent direct visual input through its Bash-only interface, but also introduces a distinct failure mode.

How the mini-swe-agent multimodal bridge works

The mini-swe-agent interface exposes Bash rather than a dedicated image-view tool, so a local /app/*.png path is not automatically converted into a model-visible image. The v3 matrix supports image-to-CAD by configuring mini-swe-agent's multimodal_regex and an explicit system template. Its essential instruction is:

When the task references a local image under /app, your first action must be
one bash call that sends that image through mini-swe-agent's multimodal
observation channel. Substitute the exact referenced path for /app/IMAGE.png:

image_path=/app/IMAGE.png; open_tag='<MSWEA_MULTIMODAL_CONTENT><CONTENT_TYPE>image_url</CONTENT_TYPE>'; close_tag='</MSWEA_MULTIMODAL_CONTENT>'; printf '%sdata:image/png;base64,' "$open_tag"; base64 < "$image_path" | tr -d '\n'; printf '%s\n' "$close_tag"

Wait for the resulting observation, inspect the image visually, and only then
continue. Derived crops may be sent through the same channel. Do not use OCR
before receiving the original image observation.

The regex recognizes the marked image_url payload and turns it into structured image content for the next model request. A custom observation template bypasses mini-swe-agent's normal 10,000-character shell-output truncation whenever the multimodal marker is present, preserving the complete data URL. In all 40 image-to-CAD trajectories for each public mini-swe-agent run, the original drawing appears as the first tool result.

The Gemini pair provides the closest view of the resulting agent difference on the 40 image tasks:

mini-swe-agent's mean advantage is 13.65 percentage points on this cohort (paired 95% CI: 5.55 to 21.75), but it uses more tokens, time, and money. The bridge enables direct visual reasoning without giving mini-swe-agent a native image-view tool; it does not make the interaction as efficient or robust as a first-class visual interface.

The bridge's limitation: marked shell errors can become images

The bridge is prompt-driven and trusts any shell output enclosed by its multimodal markers. The current command prints the opening marker before verifying that the requested file exists or that base64 succeeds. If the model names a nonexistent derived crop, the shell error can land between the markers and be forwarded as though it were encoded image bytes.

That happened in six Gemini image-to-CAD trials. The original drawing had been delivered correctly, but later the agent requested a derived PNG that had never been created. Messages such as cannot open /app/...png: No such file were placed into inline_data.data; Vertex rejected the next request because base64 decoding failed. Harbor classified four outcomes as NonZeroAgentExitCodeError and two as ApiRateLimitError, but the underlying provider response in all six was an invalid image payload. They were agent/adapter failures, not transient infrastructure failures, so all six remain zero and were not selectively retried.

This diagnosis documents a limitation of the evaluated interface; it must not become a source of benchmark-specific prompt tuning. If a future version introduces a more complicated system template, its design must not use evaluation results, trajectories, scores, or case-specific failures. The template must be motivated by a general interface requirement, frozen before evaluation, and then assessed once on the predefined held-out cohort.

Why v2 and v3 scores are separate

V3 keeps the 100-task scale but changes the task population, modalities, editing requirement, validator version, scorer behavior, network policy, Harbor version, and task schema. A v3 score is therefore a fresh benchmark measurement, not a continuation of the v2 series. V2 should remain available as a pinned historical leaderboard, while new submissions target v3.