← index
The CAD Benchmark
a benchmark that tests whether a model can reason about a mechanical part or is only matching on its shape. built on a dataset i'm making myself. first results before next year.
labs training models on mechanical design each test them on their own data. one uses scraped step files, another a supplier's parts library, another synthetic geometry it generated itself. each publishes a score, but the scores can't be compared, so nobody can say which model is better.
what's missing is a shared, held-out dataset with a scoring method anyone can run. that's what i'm building at datak, using the cad and simulation data i already sell.
| # | Model | Geometry | Edit | Mfg | Sim | Score |
|---|---|---|---|---|---|---|
| 01 | not yet published | — | — | — | — | — |
| 02 | not yet published | — | — | — | — | — |
| 03 | not yet published | — | — | — | — | — |
there are no scores yet, and i won't put placeholder numbers in a table that looks like a leaderboard. it fills in after the first run, targeted for before next year.
What it measures
a model can describe a bracket well and still not know what happens when it's loaded. the benchmark only asks questions with a known correct answer, taken from the geometry or a solver run, so a wrong answer is clearly wrong.
- geometry comprehension. identify a part's features: holes, fillets, ribs, draft, wall thickness, tolerances. answers are checked against the feature tree in the native file.
- parametric edit. change a dimension and keep the constraints valid. models that have only seen images of parts tend to fail here, because they can't tell what else moves when a hole shifts 4 mm.
- manufacturability. say whether the part can be machined, cast or moulded as drawn, and if not, which feature is the problem.
- simulation outcome. for a given load case, predict where the part fails and roughly how much it deflects. scored against the ansys, abaqus or openfoam run that came with the part.
Where the data comes from
datak already sources original cad and simulation data from a network of design partners. the parts are made for us, not scraped, and come with manifests and provenance records. the benchmark's held-out set comes from the same pipeline, so:
- it's not in anyone's training data, because it was made for this and never published.
- every task has a known correct answer from the native file or the solver run, not a human rater.
- i can keep producing new data, so if a set leaks it can be replaced.
Why i'm building it
the plan is to sell the data, run the benchmark, then train a model. the benchmark is the cheapest of the three to start with, and it shows what a good model has to do.
the model doesn't exist yet.