WeaveRL: Weaving Reconstruction into Scene-Aware Fabrics for Perceptive Reinforcement Learning

Remo Steiner*, Vikram Ramasamy*, David Tingdahl, Sam Mady, Karl Van Wyk, Nathan Ratliff, David Recasens Lafuente, Soha Pouya, Tuur Stuyck, Alex Millane
NVIDIA Corporation. Zurich, Seattle and Santa Clara.
*Equal contribution
arXiv Code (coming soon) Dataset (coming soon)
Rendered parallel RL kitchen environments
Tensorized surfel reconstruction across parallel environments

Highly parallel scene reconstruction for reinforcement learning. Rendering of RL kitchen environments for a task in which a robot places a mug in a wall-mounted cabinet (left). Our tensorized surfel reconstruction system (right) enables high-throughput mapping across thousands of parallel RL environments. Using scene-aware geometric fabrics, our method translates end-effector goals from the learned policy into collision-avoidant joint trajectories for collision-dense manipulation.

Abstract

Reinforcement learning allows robots to acquire complex skills, but producing policies for geometrically complex manipulation remains difficult. A promising approach is to learn on top of collision-avoidant controllers, such as geometric fabrics. However, these approaches have relied on static, hand-specified representations of the scene. Integrating active, online 3D perception into massively parallel RL training has so far been inaccessible. We introduce a GPU-accelerated method that reconstructs the scene as a collection of surfels across thousands of parallel simulation instances during active rollouts. This lets policies operate over sensor-derived, rather than hand-specified, geometry. On a suite of collision-dense manipulation tasks, our surfel fabrics enable policies to tackle geometrically complex scenes where primitive-based baselines fail, while maintaining sim-to-real transfer. Furthermore, policies learned with a scene-aware fabric are more robust to the introduction of novel geometry at test time, improving collision-free task completion under unseen obstacles from 35% to 61%. We release our reconstruction system, training code and test dataset to spur research in this direction.

Method

WeaveRL architecture diagram

WeaveRL Architecture. At each control step, the policy receives a batch of images from the environment and computes targets for the geometric fabric, which emits joint commands q, q̇. Batched depth frames Ftbatched are fused into surfel maps Stbatched, which the fabric queries on the next step for collision avoidance. Batched reconstruction across parallel environments enables the same perception–control loop during teacher RL training, student distillation, and deployment.

Highly Parallel Scene Reconstruction

Scaling comparison of nvblox and our batched surfel reconstruction across parallel environments

Highly parallel scene reconstruction. Scaling comparison of our approach (batched) and nvblox (non-batched; looped over environments) as the number of parallel environments increases, showing that batched mapping is substantially more efficient at larger environment counts. At 1024 environments, our approach increases throughput by 12x over nvblox (at 1cm voxel size).

Surfel Fabric for Reactive Obstacle Avoidance

Reconstruction view of surfel fabric reacting to dynamically added obstacles, with no policy driving the goal

Surfel fabric for reactive obstacle avoidance. To showcase the obstacle avoidance capabilities of surfel fabrics, we command a robot arm along a fixed left-to-right trajectory and add obstacles to the scene dynamically (left). The goal pose (shown as coordinate frame) is converted into a collision-avoidant trajectory by repulsive forces (red arrows) from the reconstruction (right).

Sim2Real Deployment

Sim2Real deployment. Surfel fabrics successfully zero-shot transfer sim2real on the Place in Box (left) and Place in Shelf (right) tasks. Built-in collision avoidance lets the policy focus on goal reaching. Primitive-based baselines fail on these collision-dense tasks, since learning both obstacle avoidance and goal-directed behavior from sparse rewards is too difficult.

Robustness to Unseen Obstacles

Success rate vs obstacle clearance for surfel fabrics, primitive fabrics, and pointcloud observation

Robustness to an unseen obstacle at test time. Policies are trained on cube pickup (no obstacles), then evaluated with an added cylindrical obstacle at varying clearance from the target object. Primitive policies must generalize from sparse-reward training alone, while our surfel fabric uses an explicit test-time scene representation for collision avoidance, increasing collision-free task completion from 35% to 61% versus primitive fabrics.

Reconstruction with repulsive forces steering the end-effector goal

Unseen obstacle at deployment. A cardboard box, absent from training, blocks the path to the target cube. Without surfel fabrics, the gripper collides with the box (left). With surfel fabrics, the task succeeds (middle); the end-effector goal predicted by the policy (shown as coordinate frame) is converted into a collision-avoidant trajectory by repulsive forces (red arrows) from the reconstruction (right).

BibTeX

@misc{steiner2026weaverl,
      title={WeaveRL: Weaving Reconstruction into Scene-Aware Fabrics for Perceptive Reinforcement Learning},
      author={Remo Steiner and Vikram Ramasamy and David Tingdahl and Sam Mady and Karl Van Wyk and Nathan Ratliff and David Recasens Lafuente and Soha Pouya and Tuur Stuyck and Alex Millane},
      year={2026},
      eprint={2609.18685},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.18685},
}