GaussianCity: A 3D City Generation Framework by NTU S-Lab

What is GaussianCity?

GaussianCity is an efficient, boundary-free 3D city generation framework developed by the S-Lab team at Nanyang Technological University (NTU). It is based on 3D Gaussian Splatting (3D-GS) technology and introduces a compact BEV-Point representation method, keeping the scene's VRAM usage at a constant level. This addresses the challenges of excessive memory and storage demands in large-scale scene generation.

GaussianCity introduces a spatially aware BEV-Point Decoder, which leverages point serialization and point transformation to capture both structural and contextual features of BEV points, generating high-quality 3D Gaussian attributes. The framework excels in 3D city generation from drone-view and street-view perspectives, achieving a 60x speedup compared to existing methods like CityDreamer, setting a new benchmark in both quality and efficiency.

Key Features of GaussianCity

Efficient large-scale 3D city generation: Quickly generates realistic, boundary-free city environments, supporting diverse views from drone perspectives to street-level scenes.

Low VRAM and storage requirements: The compact BEV-Point representation minimizes memory and storage demands, overcoming VRAM bottlenecks in large-scale scene generation.

High-quality visual effects: The spatially aware BEV-Point Decoder generates detailed and realistic 3D cities, supporting stylized editing and localized modifications.

Real-time rendering and interaction: Enables real-time rendering and interactive city generation, making it suitable for games, animations, and virtual reality applications.

Technical Principles of GaussianCity

3D Gaussian Splatting (3D-GS): Uses 3D Gaussian distributions to represent objects and structures in the scene, with GPU-accelerated rendering for efficient 3D scene generation.

BEV-Point Representation: Decomposes point information in 3D scenes into positional and stylistic attributes, leveraging bird's-eye view (BEV) and style lookup tables to compress data, ensuring VRAM usage remains constant regardless of scene scale.

Spatially Aware BEV-Point Decoder: Utilizes point serialization and transformation to capture structural and contextual information, generating 3D Gaussian attributes for high-quality scene rendering.

Efficient Rendering Pipeline: Combines positional encoders and modulated MLPs to generate 3D Gaussian attributes, using a Gaussian rasterizer to render the final image.

Project Links

GitHub Repository: https://github.com/hzxie/GaussianCity

Hugging Face Model Hub: https://huggingface.co/spaces/hzxie/gaussian-city

arXiv Technical Paper: https://arxiv.org/pdf/2406.06526

Applications of GaussianCity

Gaming & Virtual Reality (VR): Rapidly generates realistic virtual cities, supporting real-time rendering and interaction, reducing development costs and time.

Animation & Film Production: Efficiently generates complex urban scenes, supporting stylized rendering, improving production efficiency.

Urban Planning & Architectural Design: Quickly generates city layouts and building environments, aiding visualization in planning and design.

Autonomous Driving & Traffic Simulation: Creates realistic traffic scenarios for algorithm testing and traffic flow analysis.

Geographic Information Systems (GIS): Converts geospatial data into 3D visualizations, supporting urban development and analysis.