WorldCraft is a 3D world creation and customization system based on large language model (LLM) agents, introduced by the Hong Kong University of Science and Technology (HKUST). It is designed for creating and customizing realistic 3D virtual worlds. Leveraging natural language interaction, WorldCraft enables users to effortlessly generate complex indoor and outdoor scenes and fine-tune objects and layouts within these scenes.
WorldCraft consists of three core modules: ForgeIt, used for customizing individual objects with precise geometric and texture control based on procedural generation; ArrangeIt, responsible for generating scene layouts by balancing functionality and aesthetics through hierarchical numerical optimization; and a trajectory control module that allows users to control object movements and camera trajectories through dialogue, enabling the generation of animations and videos. WorldCraft is compatible with existing deep 3D generators, enriching scene assets and enabling non-professionals to efficiently realize creative designs.
Main Functions of WorldCraft
Natural Language Interaction: Users can create and customize complex 3D scenes through simple text instructions or dialogues, without requiring professional 3D modeling skills.
Individual Object Customization: Users can precisely control the geometry, textures, and materials of objects to achieve personalized designs.
Scene Layout Generation: Generates logical object layouts based on user design intentions, such as functionality and aesthetic requirements.
Scene Animation Control: Users can describe object or camera motion trajectories using natural language to generate animations and videos.
Compatibility and Extensibility: The system integrates with existing deep 3D generators (e.g., Infinigen) to enrich scene assets and enhance visual effects.
Technical Principles of WorldCraft
Coordinator Agent: Manages the entire 3D scene generation process, breaking down complex tasks into subtasks and coordinating other agent modules to complete them. Supports multi-round interactions between users and the system for real-time adjustments to generation results.
Individual Object Generation and Customization: Utilizes LLM agents to operate procedural generators (e.g., Infinigen) and generate 3D objects with precise geometry and PBR textures based on natural language instructions. A dynamic, growing manual is built through an automatic verification mechanism to guide agents in completing complex generation tasks, supporting multi-round dialogue-based editing.
Scene Layout Optimization: Models scene layout as a hierarchical numerical optimization problem, solved using simulated annealing algorithms. Users can describe layout requirements in natural language, translating design intentions into optimization goals and constraints.
Trajectory Control Module: Extends ChatCam to allow users to describe object or camera motion trajectories in natural language. Converts trajectory descriptions into specific motion commands to generate scene animations.
Deep 3D Generator Integration: The system is compatible with existing deep 3D generators, introducing artistic objects and enriching scene details. Combines procedural generation and deep generation to enhance visual quality and diversity of scenes.
Project Address of WorldCraft
arXiv Technical Paper: https://arxiv.org/pdf/2502.15601
Application Scenarios of WorldCraft
Architectural and Interior Design: Quickly generates architectural concept models and virtual showrooms, customizing indoor layouts and styles based on natural language to optimize design processes and enhance client experiences.
Film and Entertainment: Creates virtual movie sets, game levels, and complex scenes for animated films, reducing the cost of physical set construction and providing highly realistic virtual environments for VR and AR experiences.
Education and Training: Designs virtual laboratories, historical scenes, geographical environments, and medical training scenarios, enhancing learning outcomes through immersive experiences and supporting remote education and professional skill training.
Product Design and Showcasing: Rapidly generates product concept models and virtual showrooms to display designs for furniture, cars, electronic devices, and more, supporting personalized customization and the creation of virtual advertising scenes.
Culture and Art: Creates virtual museums, art exhibitions, historical buildings, and cultural festival scenes, preserving cultural heritage and providing immersive spaces for artistic creation and cultural experiences.
