Generative artificial intelligence models are capable of creating impressive 3D objects from text descriptions, but often these creations turn out to be just a beautiful shell, unsuitable for real-world use. For example, a generated cup might look like a work of art but not hold liquid. Researchers from MIT, Google, and Northeastern University have introduced InstructMesh—a tool that breaks this barrier, allowing users not only to generate but also to refine 3D models, making them functional.
InstructMesh combines the TRELLIS system for creating 3D models from text and images with the GPT-4 language model. This allows users to describe desired changes in natural language. If a generated pair of glasses model has uneven lenses, it's enough to point this out, and the system will suggest corrections. As a result, they managed to create a cup with a dragon-shaped handle, a seashell-shaped whistle, or glasses with butterfly wings. Moreover, the tool helped create a functional "bristlebot" robot in the shape of a shrimp and even a knee pad stylized as denim fabric.
The novelty of InstructMesh lies in making 3D modeling accessible to non-professionals. Tests have shown that even novices without 3D design experience were able to identify and correct errors in models with about 90% accuracy. This means that anyone can now bring their ideas to life, whether it's a personalized keychain or a device prototype, without needing to master complex software.
The developers see the future of InstructMesh in integration with augmented reality, where users can generate and print needed objects directly in their environment. They also plan to add physical simulation to predict object behavior in real conditions. This is a step towards AI generation becoming not just entertainment, but a full-fledged tool for creativity and engineering.