Skip to the yard
Waxsealai

Fine-Tuning TRELLIS.2 for More Accurate 3D Footwear Geometry

Shipped
Shipper
Robert Johnson
Readers
519

LoRA adapters improved footwear geometry on nine held-out objects, raising mean F-score from 0.276 to 0.803 while leaving texture generation frozen.

Fine-Tuning TRELLIS.2 for More Accurate 3D Footwear Geometry

Fine-tuning TRELLIS.2 with LoRA adapters substantially improved generated shoe geometry in a small evaluation: mean F-score rose from 0.276 to 0.803 across nine held-out objects. The experiment targeted two stages responsible for structure and shape, while keeping the material-generation stage frozen. That choice reflects the central practical lesson: tune the components tied to the task, and make sure the training data matches what each component is expected to learn.

Stage 1

The work uses 188 footwear objects drawn from the 1,030 objects in Google Scanned Objects (GSO). The pretrained model generally generated shoes that looked plausible, but their soles were often inaccurate. The aim was to improve that geometry with a compact set of trainable parameters, then compare outputs against objects excluded from training.

Why TRELLIS.2 can use targeted adapters

3D generation has to contend with different file formats and scene representations. GLB and PLY are examples of file formats; neural radiance fields and 3D Gaussian splatting are examples of ways to represent 3D content. Each comes with different tradeoffs. TRELLIS introduced a unified latent representation that can be decoded into different output representations, reducing the need to rely on separate generation models for each format. The TRELLIS paper describes that approach.

TRELLIS.2 builds on this family of models with an improved representation of 3D structure. Yet a general-purpose model can still miss details important to a particular use case. Here, the visible weakness was the shoe sole. The experiment asks whether training on footwear examples can improve that part of the output without retraining the full model.

O-Voxel

TRELLIS.2 processes an image through several stages. BiRefNet removes the background, and DINOv3-L encodes the remaining image into feature tokens that condition generation. A sparse-structure flow diffusion transformer (DiT) predicts a coarse shape. Its latent has dimensions of 16³ × 8; a variational autoencoder (VAE) decoder converts it into a 64 × 64 × 64 occupancy grid, where each voxel indicates whether a location belongs to the object.

The next stage generates detailed geometry. TRELLIS.2 uses O-Voxel, a sparse voxel representation that retains voxels on an object's surface. Those voxels can store local geometric information, such as mesh vertices, together with material attributes. The format supports conversion between meshes and O-Voxel in seconds on a CPU or GPU, according to the description of the pipeline.

The shape flow DiT combines the occupancy grid with the image features to produce a latent, which another VAE decoder turns into O-Voxel geometry at 512 × 512 × 512 resolution. The texture flow DiT then uses the shape latent and image features to predict surface materials. Its decoder produces base color, metallic, roughness, and alpha values for each voxel. The pipeline assembles geometry and material attributes into a textured mesh and exports a GLB file.

Preparing LoRA Adapters

TRELLIS.2 also uses structured sparse latents. The shape and texture flow DiTs operate on sparse tensors, processing approximately 10% of the voxels in the grid rather than spending computation on empty space. The sparse-structure flow DiT is dense. This division matters when deciding where adapters belong: the first model guides coarse structure, the second determines detailed shape, and the third handles materials.

Why the texture model stays frozen

The experiment adds LoRA adapters to the sparse-structure flow DiT and the shape flow DiT, since those stages control the geometry under study. It leaves the texture flow DiT unchanged. The GSO subset provides diffuse color maps, but not the separate material maps needed to train the texture stage appropriately. Diffuse maps combine surface color with baked lighting, so using them as material targets could teach the model to treat lighting effects as surface properties.

There is a second reason for restraint: a small, mismatched dataset could cause catastrophic forgetting, where fine-tuning weakens capabilities the pretrained model already has. In this case, the existing model was already producing acceptable colors and textures. The available data and the problem both point toward adapting geometry alone.

Training Loop

LoRA, or low-rank adaptation, trains small adapter matrices while the original model weights remain frozen. That reduces the trainable parameter count and the memory required for gradients and optimizer state compared with full fine-tuning. The implementation uses Hugging Face's PEFT library. Across the two targeted DiTs, the configuration trains approximately 9.3 million adapter parameters.

The sparse-structure DiT uses rank 8 adapters in its attention and feed-forward (MLP) layers. The shape DiT uses rank 16 adapters in its attention layers. Rank sets the size of the adapter matrices and therefore how much capacity they have to learn. These settings are specific to this experiment; they are not evidence that the same ranks will suit other object categories or training sets.

Preparing data and running training

The workflow packages dependencies into two Docker images: one for preprocessing and another for fine-tuning. The containers can run on Hugging Face Jobs or on a local machine with a suitable GPU setup. The scripts accept arguments to select a training mode, configure run length, and optionally upload adapter checkpoints to the Hub. The companion repository contains the scripts and configuration files; prepared footwear latents are also available as a dataset.

Loss

Preprocessing uses Blender to render 16 views of each object. It runs without a display through xvfb-run, and the objects are also processed through the TRELLIS.2 data toolkit. Frozen TRELLIS.2 encoders convert the meshes into training latents, which are saved so subsequent runs can reuse them. The preprocessing run for all 188 objects took approximately one hour on an A100-large job. A compact version of the preprocessing script is available in the repository.

Preprocessing

For evaluation, nine objects are held out, leaving 179 for training. Each training example uses one randomly selected rendered view as its conditioning image. The held-out objects let the team check whether the adapters improve generation for footwear absent from the fine-tuning examples.

The training container loads the prepared dataset and a preset or configuration file. Its two modes, “ss” and “shape,” select the sparse-structure flow DiT or the shape flow DiT, respectively. Each stage is trained in a separate run. Metrics are recorded with Trackio, and a dashboard can be used to monitor progress.

Training uses a batch size of 4 with 4 gradient accumulation steps, giving an effective batch size of 16. The adapters are inserted with PEFT's inject_adapter_in_model() function. TRELLIS.2 has a custom forward pass, so the standard PeftModel() wrapper does not fit its interface. Adapter weights remain in 32-bit floating point (fp32), while bfloat16 (bf16) autocasting reduces memory use for some forward-pass operations. Autocasting does not change the storage precision of the frozen backbone weights.

The optimization objective is rectified flow matching, the same objective used during TRELLIS.2 pretraining. During each training batch, DINOv3 encodes the conditioning image. Classifier-free guidance dropout sets conditioning to zero for a randomly selected fraction of samples, controlled by the p_uncond setting. The loop calculates flow-matching loss, accumulates gradients over four batches, clips gradients, and updates the adapter weights. Gradient checkpointing reduces GPU memory use by recomputing some intermediate values during backpropagation.

The loop logs loss every 10 optimizer steps and saves an adapter checkpoint every 500 steps. These details make the scripts adaptable, but the training loss alone cannot establish whether generated geometry has improved. That requires comparing model outputs against held-out reference scans.

Results and limits

Each run used one A100 with 80 GB of memory on Hugging Face Jobs. Training the sparse-structure adapter took 7.2 hours; training the shape adapter took 3.6 hours. The sparse-structure training loss dropped quickly at first and then more gradually, while the shape-stage loss stayed at approximately the same level. Those curves describe the optimization objective, not output quality.

On five of the nine held-out objects shown in the comparison, the footwear adapters produced soles closer to the reference scans. Across all nine objects, the combined adapters raised mean F-score from 0.276 for the base model to 0.803. Chamfer distance fell from 0.174 to 0.033, and normal consistency rose from 0.530 to 0.804. The F-score improved for each of the nine held-out objects.

Object Metrics

These figures are encouraging for this footwear experiment, but the evaluation set contains only nine objects from one category. The results establish a strong gain on those examples; they do not show how well the adapters transfer to other object types, datasets, or generation tasks. The training losses also cannot answer that question. The reported Trackio runs provide additional statistics.

The adapters are available at suvadityamuk/trellis-2-finetune-gso. For practitioners, the useful result is both the measured geometry improvement and the method behind it: prepare reusable latents, hold out objects for evaluation, and adapt only the pipeline stages that correspond to the known failure. Applying the same recipe to models such as Pixal3D, UniLat3D, or Gaussian-splat and world-generation systems may require different representations and training procedures. The next test is whether this component-by-component approach continues to help when the target category and data change.

Source: Robert Johnson · huggingface.co

Same vesselRelated cargo