PROPOSED PROTOCOL · VERSION 0.1
A useful BIM benchmark asks whether the task was completed, whether the result works inside the application, and what it took to get there.
This is a proposal for future tests. The first pilot has PDF evidence and token notes; it has not been scored under this protocol.
COMMUNITY / BLIND COMPARISONS
The rating page presents two full-sheet previews from the same experiment, with the model, effort, token usage and original filenames omitted. Pair assignment and A/B order are selected on the server. Less-reviewed pairs receive priority. The publicly browsable originals mean this reduces casual identity bias; it cannot prevent determined recognition.
Counted votes require a Google account with verified email, a valid session, same-origin submission, a server-verified Turnstile challenge tied to the assigned pair, and at least 20 seconds since assignment. Each account receives one active pair per experiment version and may submit each unordered pair once, up to 20 comparisons per experiment version. Cannot-judge consumes a comparison but remains separate from preference outcomes. Refreshing or opening a second tab returns the same active pair.
The database enforces duplicate rejection and the contribution cap in the write transaction. Account and network request limits add protection. An expired pair or old experiment version cannot receive a vote. Accounts are not people: multi-account manipulation and coordinated humans remain possible. Viewing time is a minimum waiting period, not proof of attentive review.
Community preferences are stored separately from expert PDF rubric scores and native BIM checks. No public community leaderboard is produced by this first implementation. Before ranking, define the aggregation method, voter coverage and uncertainty policy, then publish those choices with reviewer counts. The owner can mark suspicious votes held or excluded, preserving an audit reason; disagreement with the majority alone is not grounds for exclusion.
While voting setup is incomplete, the page explicitly offers previews only. Previews do not store votes, do not establish identities and do not affect results.
PILOT / PROPOSED RATING
GPT-6 Astra · Light and GPT-6 Astra · Ultra are different configurations. Review and rank each configuration separately. Do not pool five different effort settings as if they were five repeated trials of one model.
For this eaves pilot, two independent architectural reviewers could score four criteria from 0–5. Each criterion contributes up to 25 points: score × 5. Average the reviewers’ completed totals and publish both individual scores, rubric version, reviewer count and short evidence notes. This is a proposed visual-review score; none has been assigned yet.
| Criterion | Points | What the PDF can show |
|---|---|---|
| Visible brief coverage | 25 | Depiction of the requested eaves, double plate, framing and batting; visible dimensions and labels. Family identity and actual dimensions remain unverified. |
| Junction coherence | 25 | The depicted bearing and birdsmouth, connections, and cavity continuity are intelligible. This is visual coherence, not a measured or engineering approval. |
| Drawing legibility | 25 | Readable geometry, line hierarchy, useful framing and absence of distracting overlaps. |
| Annotation clarity | 25 | Readable text, clear leaders and dimensions, with no obvious contradictory labels. |
Suggested anchors: 0 = absent or unusable; 1 = major problems; 2 = substantial corrections needed; 3 = usable with corrections; 4 = strong with minor issues; 5 = clear and complete for the criterion. Missing evidence is unverified, not zero. Publish an overall score only when all four criteria are reviewed.
Hide the model name, effort and token use during review, randomize the order, and use consistent PDF viewing conditions. Establish detailed checklist anchors and reviewer agreement before publishing ranks. Blind A/B preference can supplement this, following the approach described by AA-3D; preference and rubric scores should remain separate.
Native Revit correctness, execution compliance and reliable revisions are a separate audit. The PDF score is not an overall BIM score. Token usage stays in its own graph; quality versus usage becomes meaningful once completed reviews and comparable resource scopes exist.
BIM / THE FOCUS
AA-3D evaluates interactive three.js browser scenes across 55 tasks, using blind human preferences for its Elo rankings. It is a useful reference for visual evaluation; its published protocol does not assess native BIM authoring.
BimBench’s proposed focus is professional BIM workflows: measurable geometry, native elements and families, useful parameters, coordinated documentation, quantities, and reliable revisions. A 2D drafting task belongs in that scope alongside 3D modelling. Visual review is one dimension; native files and execution traces supply the technical evidence. The present pilot supplies PDFs and token notes only.
01 / WHAT WE HAVE
The first experiment contains one PDF for each of 15 model and effort configurations, plus a supplied token note. The prompt requests a 1:10 Revit drafting view of a timber eaves junction created through pyRevit MCP. No repeated trials, native RVT files, complete execution logs, run times, or billing records were supplied for this website.
The PDFs support inspection of the visible results. Dimensions printed as text are not proof of the underlying geometry. A drawing that resembles a standard family is not proof that a standard family was used. Memory restrictions, computer-use restrictions and opening the view before modelling require execution evidence.
The public display mapping is SOL → GPT-5.6 Sol, ASTRA → GPT-6 Astra, and SOL 6 → GPT-6 Sol. These names were checked against GPT-5.6 Sol documentation, GPT-6 Astra documentation and GPT-6 Sol documentation on 17 October 2026. The mapping is inferred from aliases and export chronology; exact historical model snapshot IDs remain unverified. Original aliases and filenames are retained in the dataset.
Input includes cached input. Output includes reasoning output. Neither subset is added again. Missing values remain unreported. Where input is missing but total and output are present, input is calculated as total minus output and marked as derived. The SOL Ultra total is calculated as input plus output.
GPT-6 Astra Ultra reports helper-agent usage within its 3,447,140 total. GPT-6 Sol Ultra reports 7,119,141 for the chat and a separate 11,514,922 including delegated agents, with 11,168,384 cached input tokens in that combined count. That combined cache count must not be assigned to the smaller chat-only total. Agent scope is unspecified for the remaining runs.
Totals are conversation snapshots. They are not final billing measurements and do not establish cost efficiency. The pilot does not estimate dollar costs or rank quality.
Original token note · Structured dataset with PDF SHA-256 checksums
02 / WHAT TO MEASURE
Publish these independently before considering a composite index. Define the checklist, tolerances and grading anchors before running models. Score missing evidence as “unverified”, rather than silently treating it as a pass or a failure.
| Dimension | Proposed measure | Required evidence |
|---|---|---|
| Brief completion | Pass / fail / unverified for each explicit requirement; publish passed checks and the complete checklist. | Prompt, final artifact, execution trace. |
| Geometric fidelity | Measure pitch, horizontal overhang, framing depth and contact at the bearing against predefined tolerances. | Native geometry audit and saved model. |
| Native usability | Check real detail components, family/type identity, native batting, editable dimensions and a controlled revision. | Native element inventory and edit/reopen test. |
| Drawing quality | Blind expert review: clarity, hierarchy, annotation legibility and drawing coordination, each on an anchored 1–5 scale. | Randomized, consistently exported sheets; at least two independent reviewers. |
Do not reward extra layers, decorative annotations or a preferred roof orientation unless the task calls for them. The original brief leaves some construction choices open; document these ambiguities before future trials. This measures task performance and drawing usability, not building-code approval.
03 / A FAIR COMPARISON
For a future dashboard, show completion rate and quality next to elapsed time and billed cost per successful task. A quality-versus-cost scatter plot becomes useful once the underlying measurements share the same accounting scope. Tokens alone are not cost.
04 / A PRACTICAL TEST SUITE
Repeat the eaves detail from a fixed family library. Save the native file, reopen it and inventory element classes, families and types. Verify the drawing uses editable components rather than an imported raster or a single flattened graphic.
Give a completed baseline model a new roof pitch and overhang. Check the requested changes, bearing contact, insulation, annotation updates and unaffected elements. Use a baseline diff to measure unintended edits.
Create a plan, section and sheet from a fixed small model. Check view scales, placements, tags and annotation collisions. Add a quantity takeoff task with expected element counts and totals computed from the baseline.
Detect planted geometric clashes in a synthetic model, identify the correct elements and propose or execute a bounded correction. In a separate task, introduce a recoverable tool error and measure whether the agent finishes without duplicate or damaged elements.
Adapt the same task intent to applications such as Archicad, BlenderBIM/Bonsai and Veido after a working tool connector exists. Keep application-specific native scores separate. Compare exported geometry, units, object identity and property retention using a declared IFC exchange profile.
Make a small reusable family or object with three parameters. Exercise a predetermined parameter matrix and check geometry, constraints, naming and valid output across every case.
These are proposed experiments. No results for these tasks or additional applications are claimed.
05 / MAKE RESULTS AUDITABLE
Publish a versioned task definition, environment manifest, original prompt, native artifact where permitted, PDF export, run trace, structured resource use, evaluation checklist and reviewer notes. Attach hashes and stable run IDs. Keep corrections in a changelog instead of overwriting a published score.
Use bimbench.com as the concise primary address and redirect bimbenchmark.com to it when public hosting is configured. This local demo does not configure either domain.
The presentation follows the supplied Veido website palette. The longer-term dashboard idea is inspired by Artificial Analysis, which presents separate capability, speed and cost comparisons. BimBench proposes adapting that separation to inspectable BIM artifacts; it does not reuse their scores or claim an affiliation.