Divyansh Chauhan

The CAD Benchmark

a benchmark that tests whether a model can reason about a mechanical part or is only matching on its shape. built on a dataset i'm making myself. first results before next year.

labs training models on mechanical design each test them on their own data. one uses scraped step files, another a supplier's parts library, another synthetic geometry it generated itself. each publishes a score, but the scores can't be compared, so nobody can say which model is better.

what's missing is a shared, held-out dataset with a scoring method anyone can run. that's what i'm building at datak, using the cad and simulation data i already sell.

The board↓ highest first
#ModelGeometryEditMfgSimScore
01not yet published—————
02not yet published—————
03not yet published—————

there are no scores yet, and i won't put placeholder numbers in a table that looks like a leaderboard. it fills in after the first run, targeted for before next year.


What it measures

a model can describe a bracket well and still not know what happens when it's loaded. the benchmark only asks questions with a known correct answer, taken from the geometry or a solver run, so a wrong answer is clearly wrong.

410.0 115.0 GEOMETRY: name the feature EDIT: move it, keep constraints MFG: is the fillet moulda­ble SIM: where does it yield PART 0001 · STEP AP242
Fig. 01 one part, four questions. a model can pass the first and fail the other three, and current evaluations don't separate these. diagram, not a dataset sample.

Where the data comes from

datak already sources original cad and simulation data from a network of design partners. the parts are made for us, not scraped, and come with manifests and provenance records. the benchmark's held-out set comes from the same pipeline, so:

status: dataset in progress, scoring harness in progress, no results yet. first public run targeted before next year. if you're training on mechanical design data and want to be on the first board, email me.

Why i'm building it

the plan is to sell the data, run the benchmark, then train a model. the benchmark is the cheapest of the three to start with, and it shows what a good model has to do.

the model doesn't exist yet.


back to index what happened at blip