MeshPad - A Sketch-Driven AI 3D Mesh Generation and Editing Tool

What is MeshPad?

MeshPad is an interactive artistic mesh generation and editing tool based on sketch input, capable of quickly transforming simple 2D sketches into high-quality 3D mesh models with real-time editing support. Users can add or remove lines on the sketch to modify the 3D mesh, such as deleting regions or adding new geometric shapes. It employs a triangle sequence representation and uses a Transformer model to capture complex geometric relationships, generating high-quality meshes. A speculative prediction strategy significantly reduces computational costs, speeding up the editing process, with each edit taking only a few seconds. The quality of the generated meshes improves by 22% in Chamfer distance compared to existing methods.

Main Features of MeshPad

Sketch-Driven 3D Mesh Generation: Users only need to draw simple 2D sketches, and MeshPad automatically converts these sketches into high-quality 3D mesh models.

Real-Time Interactive Editing: MeshPad supports real-time editing of the generated 3D meshes. Users can add, delete, or modify lines on the sketch, and the changes are immediately reflected in the 3D mesh model.

Efficient Mesh Processing: MeshPad uses a triangle sequence representation to efficiently handle mesh addition and deletion operations. The built-in Transformer model captures complex geometric relationships, generating high-quality meshes.

High-Quality Mesh Output: The meshes generated by MeshPad improve by 22% in Chamfer distance compared to existing methods, resulting in more accurate 3D models with richer details.

Technical Principles of MeshPad

Triangle Sequence Representation: MeshPad uses a triangle sequence representation to efficiently handle mesh addition and deletion operations.

Transformer Model: MeshPad is based on a large Transformer model to capture complex geometric relationships. The Transformer model recursively generates new triangular meshes during mesh addition and deletion operations, seamlessly integrating the newly generated meshes with the existing model.

Add Operation: The Transformer uses an autoregressive model to predict the position of new triangular faces, ensuring perfect integration of the new parts with the original mesh through "speculative prediction" alignment of vertex coordinates.

Delete Operation: A classification network analyzes the vertices in the mesh, providing "delete labels" to precisely remove unwanted parts without damaging other structures.

Speculative Prediction Strategy: MeshPad introduces a vertex-aligned speculative prediction strategy. By predicting multiple output tokens for vertices, it significantly reduces computational costs and speeds up the editing process. Each edit takes only a few seconds, achieving a true real-time interactive experience.

Real-Time Interaction and Optimization: MeshPad's interactive design supports real-time editing of 3D mesh models through simple sketch operations (such as adding or deleting lines).

Project Address of MeshPad

Project Website: https://derkleineli.github.io/meshpad/

Technical Paper: https://arxiv.org/pdf/2503.01425

Application Scenarios of MeshPad

Art Design: Quickly convert sketches into 3D sculptures, animations, or game models.

Architectural Design: Transform hand-drawn sketches into architectural models, allowing real-time adjustment of design details.

Industrial Design: Generate and modify product prototypes, accelerating the design iteration process.