May 12, 2025 15:00:00

The model 'LegoGPT' that generates physically stable Lego block structures from text and the dataset 'StableText2Lego' are now available

A research team at Carnegie Mellon University has announced ' LegoGPT ', which can generate Lego-built structures from a text prompt entered by the user. The research team has also released ' StableText2Lego ', a dataset containing over 47,000 Lego structures and over 28,000 unique 3D objects.

Generating Physically Stable and Buildable LEGO Designs from Text

https://avalovelace1.github.io/LegoGPT/

AvaLovelace1/LegoGPT: Official repository for LegoGPT, the first approach for generating physically stable Lego brick models from text prompts.
https://github.com/AvaLovelace1/LegoGPT/

The published research is based on StableText2Lego, a large-scale Lego dataset that contains over 47,000 Lego structures and over 28,000 3D objects, with each structure paired with a 'stability score' indicating its physical stability and a text description.


The research team first voxelized the 3D mesh from ShapeNetCore , a database of 3D models, into a 20x20x20 grid and then converted it into a Lego block layout using a process called 'legolization.'

The layout of the blocks was then randomized to create multiple structural variations for each shape while preserving the overall shape, with a check for physical stability and unstable designs filtered out.

Based on StableText2Lego, which was built using the Lego structure data obtained in this way, the research team fine-tuned it using

Llama-3.2-1B-Instruct as the base model.

The research team pointed out that

LDraw , a 3D graphic file format for modeling LEGO works, has the disadvantage that the dimensions of the blocks are not explicitly stated and that it contains information unnecessary for this task, such as the orientation and scale of the blocks. Therefore, each LEGO block is represented in a unique format of '{height} x {width} ({x coordinate}, {y coordinate}, {z coordinate})'. This significantly reduces the number of tokens required to represent the design while still including the dimensional information of the blocks that is essential for 3D inference.

Even when trained on physically stable data, the model can generate designs that violate physical laws or assembly constraints. Therefore, the research team ensures the stability of the generated LEGO structures by incorporating physical stability verification during the model's autoregressive inference. They also apply detailed textures to the generated shapes and assign uniform colors to individual blocks.

You can see what kind of LEGO structure is actually output in response to the text prompt by watching the movie below.

'A classical guitar'


'A basic sofa'


Add texture to this basic sofa with the prompt 'Vintage floral tapestry with deep reds and golds' to create this look.


'A streamlined vessel with a long, narrow hull.'


And the following movie shows how the LEGO structure output by this 'streamlined ship with a long hull' is actually assembled by a robot.


The StableText2Lego dataset and the LegoGPT code and models are publicly available on GitHub. However, because Llama-3.2-1B-Instruct, on which LegoGPT is based, is a gated model, access requires a Hugging Face user access token . In addition, a Gurobi license is required to perform stability analysis.

A demo of LegoGPT is available on Hugging Face and can be accessed below.

LegoGPT-Demo - a Hugging Face Space by cmu-gil
https://huggingface.co/spaces/cmu-gil/LegoGPT-Demo