UrbanWind-AgentBench · WindPilot
A benchmark of 200 urban wind CFD tasks with recomputable two-layer scoring, and an execution framework that ties every reported metric back to raw solver evidence.
Live field · task-061 · OpenFOAM v2412 · iteration 400 · z = 0.44 m
Watch · 1 min 24 s
The benchmark, the execution framework and a replay of a real run on task-061. Captions are in Chinese.
Terminal output, flow fields and scores in the video come from recorded runs; the replay is trimmed and sped up.
01 · Key finding
On 30 OpenFOAM tasks with independent references, the base agent reached the full structural score on 28 tasks, yet delivered metrics that could be recomputed from raw outputs on only 6.
With the same model name and budget, WindPilot raised that to 17. The deliverables looked complete in both cases; the difference is whether the reported numbers trace back to actual solves and samples.
AI agents based on large language models (LLMs) can now configure and run computational fluid dynamics (CFD) solvers, but a completed run does not ensure that delivered metrics can be recomputed from raw outputs. This gap was examined for urban wind simulation, where multi-case comparisons must hold non-target conditions fixed and reported values must be traceable to raw samples. UrbanWind-AgentBench, comprising 200 tasks and 367 cases in 10 engineering task families, was constructed. Public task contracts specify variable factors, fixed conditions, numerical thresholds and deliverable requirements, and two-layer scoring separates a structural score for deliverables, run provenance and raw evidence from a numerical score obtained by independent recomputation against references. WindPilot was then proposed as an execution framework that incorporates contract parsing, execution provenance, deterministic verification and bounded acceptance review into the LLM execution loop. On 30 OpenFOAM tasks with independent references, at least the structural maximum of 60 points was reached by a base agent on 28 tasks, but numerical credit was obtained on only 6. With the same model name and budget, WindPilot increased the number of tasks with numerical credit to 17, full-score tasks from 0 to 3 and the mean task score from 62.5 to 72.5. Mean action steps fell from 130.4 to 104.9, and mean token consumption decreased by 12.9%. The mean gain for atmospheric boundary layer (ABL) tasks was 18.5 points. A completed simulation run is therefore not sufficient for a usable urban wind result; usability depends on checks that tie reported metrics to raw evidence during execution.
02 · UrbanWind-AgentBench
Each family has 20 tasks. Every engineering stage — materials, geometry, meshing, boundaries, solution and interpretation — has an explicit entry point, and outputs continue to inform later stages.
Coloured by family. Outlined tiles are the 30 OpenFOAM tasks with independent references used in the evaluation. Hover for details.
task —
03 · Contracts & two-layer scoring
Define the comparison, e.g. three aerodynamic roughness lengths in task-093.
Preserve comparability: inlet speed, direction, mesh and turbulence model stay unchanged.
Public checks, e.g. wind-speed drift ≤ 5% and TKE drift ≤ 2% over six 60 s windows (task-087).
Six common files plus task-specific arrays, so results can be read programmatically.
design.jsoncase_definition.jsonrun_manifest.jsonmetrics.jsonevidence_manifest.jsondecision.md
A score of 60 means complete deliverables with zero numerical credit. Only when reported values can be recomputed from raw data does a task exceed 60. Numerical references are uploaded to the scoring environment after the agent finishes, so they are never visible during execution.
04 · WindPilot
A harness is the program around the LLM that decides what the model sees, which tools it calls, when checks occur and when execution stops. WindPilot organises four checks outside the model so that feedback points to actual configurations and deliverable files.
| Compute | AMD EPYC 7B12 · 128 physical cores · 16 cores per task · OpenFOAM v2412 + MPICH |
|---|---|
| Model | DeepSeek-V4.1-Flash · official API · default sampling (execution and acceptance review) |
| Budget | ≤ 150 action steps · last 15 rounds kept · 5 h + 2× historical solver time · 300 s foreground limit |
| Review | Delivery checks from step 90 · ≤ 2 review calls per task · evidence view ≤ 96 KiB |
| Runs | One run per task per condition · independent scoring ≤ 900 s, outside the agent budget |
05 · Results
30 OpenFOAM tasks with independent references (13 ABL, 16 RANS/LES, 1 meshing). One run per task per condition; the base agent and WindPilot use the same model name.
06 · Agent traces
A replay of the recorded WindPilot run on task-061 (neutral log-law inlet around a D×D×4D building). Commands and outputs are taken from the trace and trimmed for length.
Review found a ground heat-flux boundary inconsistent with the contract. The agent adjusted near-wall boundaries, reran all three cases, rebound run records and supplied the post-processing script, statistical window and hashes before submitting at step 130.
A budget-triggered check at step 91 found velocity arrays of shape 52×3 instead of the raw 20×52×3 — the time axis was lost. After re-exporting from raw samples, all nine checks passed at step 99 and stayed clean at steps 116 and 120.
101 deduplicated errors in 10 tasks: 62 missing or unreadable files, 15 raw-data mismatches, 14 stale hashes, 7 incomplete decision notes. 97 (96.0%) were absent from final verification. Programmatic checks took 20.8 s per task on average.
07 · Citation
@article{urbanwind_agentbench_2026,
title = {Does a completed simulation run mean a usable urban wind result for AI agents?},
author = {TODO},
journal = {Under review},
year = {2026}
}