LegoGPT generates a LEGO structure from a user-provided text prompt in an end-to-end manner. Notably, our generated LEGO structure is physically stable and buildable.
Abstract
We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models.
Step by step generation of LEGO structures from text
"A streamlined vessel with a long, narrow hull"
"A classical guitar"
"A basic sofa"
"A bookshelf with horizontal tiers"
"A high-backed chair"
"A backless bench with armrest"
Automated assembly of generated LEGO structures using robots (8x speed)
"A streamlined vessel with a long, narrow hull [...]"
"An asymmetrical six-string guitar [...]"
Generated Textured LEGO Models
"Rustic stone bench with moss growth [...]"
"Hot rod with flame paintwork [...]"
"Rustic farmhouse chair built from reclaimed wood [...]"
"Live edge walnut table [...]"
"Comfortable lounge chair wrapped in Japanese shibori fabric [...]"
"Cyberpunk holographic material with neon purple and blue gradients [...]"
"Rustic farmhouse armchair built from reclaimed wood [...]"
"Vintage floral tapestry with deep reds and golds [...]"
"Gothic cathedral bookshelf with arch details, medieval style [...]"
"Japanese sliding bookcase with shoji screens, traditional design [...]"
"Victorian library shelving with carved moldings [...]"
Generated Colored LEGO Models
"Parlor guitar with ladder bracing [...]"
"Electric guitar in metallic purple [...]"
"Steel resonator with engraved body[...]"
"Sunburst Les Paul with amber finish [...]"
English (US) ·