Local LLMs · NVIDIA Isaac Sim
From language to executable 3D edits
“Make the big gear gold.” A natural-language interface for working with existing 3D scenes.
Original captures. Open either image to inspect the full-size scene.
The problem
A request such as “make the big gear gold” leaves several decisions implicit: which object is meant, which property should change, and what operation the scene tool expects. The extension connects these language-level choices to concrete objects and operations in an existing USD scene.
My contribution
My work spans the Isaac Sim interface, scene integration, structured action pipeline, and comparison of local language models. I connected scene discovery to object selection, material and transform operations, and short conversational follow-ups.
What makes the interface work
The model receives a catalog of available scene objects. Pydantic schemas, JSON extraction, and retry logic constrain its proposed actions before execution. A manual mode provides direct control, while feedback handles unknown objects and unsupported requests.
What the prototype demonstrates
The archived scene sequences show material, position, rotation, and lighting edits applied in Isaac Sim. The project demonstrates language-guided manipulation of existing geometry, with a small model comparison to examine correctness and latency.
A small model comparison
Ten scripted cases from an earlier version of the agent pipeline, recorded in January 2026. The cases check object selection, valid operations, and handling of unsupported requests.
| Model | Cases passed | Mean time |
|---|---|---|
| GPT-OSS-20B | 10 / 10 | 5.88 s |
| Qwen2.5-7B | 9 / 10 | 1.04 s |
| Mistral-7B | 8 / 10 | 1.73 s |
| Phi-3-Mini | 8 / 10 | 1.01 s |
An exploratory comparison on this small test set. These measurements describe the earlier agent pipeline; they are not a benchmark of the full editing extension or general model accuracy.