Researchers from the Technical University of Munich (TUM) and Politecnico di Torino, in collaboration with Audi AG, developed MeshGPT, a 3D mesh generation software that uses generative AI. Their innovative approach utilizes decoder-only transformers to create triangle meshes that are cleaner and more coherent, boasting a remarkable 9% increase in shape coverage and a 30-point boost in FID scores (a metric used to assess the quality of images created by a generative model), compared to existing methods. This means sharper edges, higher fidelity, and compact designs that set a new quality standard. The full paper is available here.
MeshGPT creates triangle meshes by autoregressively sampling from a transformer model that has been trained to produce tokens from a learned geometric vocabulary. These tokens can then be decoded into the faces of a triangle mesh. MeshGPT reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields. Inspired by recent advances in powerful large language models, the researchers adopted a sequence-based approach to autoregressively generate triangle meshes as sequences of triangles.
The first step was to learn a vocabulary of latent quantized embeddings, using graph convolutions, which inform these embeddings of the local mesh geometry and topology. The term “quantized embeddings” refers to a technique used in natural language processing (NLP) and machine learning to represent and store word embeddings or continuous-valued vectors in a quantized, discrete form.
These embeddings are sequenced and decoded into triangles by a decoder, ensuring that they can effectively reconstruct the mesh. A transformer is then trained on this learned vocabulary to predict the index of the next embedding given previous embeddings. Once trained, this model can be autoregressively sampled to generate new triangle meshes, directly generating compact meshes with sharp edges, more closely imitating the efficient triangulation patterns of human-crafted meshes.
MeshGPT demonstrates a notable improvement over state-of-the-art mesh generation methods, with a 9% increase in shape coverage and a 30-point enhancement in FID scores across various categories. Existing approaches often either miss details produce over-triangulated meshes, or output too simplistic shapes. The MeshGPT method can also be used to generate 3D assets for scenes.
Study authors include: Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, Matthias Nießner
