Sim-to-sim VLA fine-tuning

VLA data without teleoperation

Vision-language-action models need robot-specific demonstrations, but human teleoperation is slow, expensive, and hard to scale to every custom scene. This work replaces that bottleneck with simulation plus classical motion planning: a planner generates success-gated demos, π0.5 is fine-tuned on them, and both teacher and student are evaluated under the same interface and success criterion.

Gap: Teleop-centric VLA data is costly and poorly suited to custom robot and scene deployments
Approach: MoveIt in Isaac Sim → success-gated demos → π0.5 LoRA → redeploy on the same joint-trajectory interface
Data: 350 successful MoveIt pick-and-place examples (failures discarded)
Result: ~60% task success after LoRA fine-tuning in the same sim scenes

Seeded sim → Planner demos → Success gate → Fine-tune → Same-interface deploy

Videos

Baseline

Baseline — π0.5 without fine-tuning; fails on this sim stack (camera setup, control convention, and related distribution shift)

After fine-tuning

Successful action after fine-tuning — example 1

Successful action after fine-tuning — example 2

Successful action after fine-tuning — example 3

Successful action after fine-tuning — example 4

What existing approaches lack

Most VLA fine-tuning pipelines still treat human teleoperation — or large third-party demo corpora — as the main source of robot-specific behavior. Every new kinematics, workspace, or custom scenario then needs another teleop setup. Success criteria are often soft, and the learned policy is rarely compared to a teacher under the same control contract, so progress is hard to measure honestly.

Simulation is often used as a dump of trajectories rather than a disciplined factory for demonstrations a classical planner already knows how to execute well.

Approach

A MoveIt 2 expert (sampling-based planning, IK, time-optimal trajectories) repeatedly runs a language-conditioned tabletop task — Franka Panda: pick up the ball and place it in the bowl — across seeded scene variations. Those rollouts become the training set: no human in the loop, and the actions already respect the robot’s kinematics and collision constraints.

Only successes are kept. Success means the arm picks the ball from the table and places it so the ball stays in the bowl. Collisions, failed grasps, slips during transfer, or a ball that leaves the bowl are discarded. The current set has 350 such examples.

Planner and policy are peers behind one thin interface — joint states out, joint trajectories in — so π0.5 (OpenPI, LoRA) redeploys as another planner on the same bus, in the same scenes, under the same success criterion. Observations record what the cameras measured; action labels record what the planner commanded, with sim-time alignment so image–action pairs stay trustworthy.

A natural extension targets custom real-world scenes without a teleop station: capture the environment with an RGB-D camera, reconstruct it in the simulator, then run classical planning to generate demos that fit the target kinematics.

Baseline vs fine-tuned results

Without fine-tuning, base π0.5 fails consistently on this stack. The pretrained model is trained on diverse corpora that include Franka among other robots, but that breadth is not plug-and-play: training mixes platforms, control conventions, and camera setups, while deployment uses our Isaac Sim cameras and absolute joint trajectories on a ROS interface. Camera intrinsics and extrinsics also change from application to application, so fine-tuning is the norm for most use cases — not optional polish.

After LoRA fine-tuning on the planner-generated successes, task success rises to roughly 60% under the same criterion. That is not saturation, but it shows teleop-free sim data can adapt a general VLA to a specific robot and scene.

Takeaway

Planner-generated, success-gated simulation data — and eventually RGB-D scene reconstruction — is a practical path to adapt VLAs without teleoperation.

Keywords

Isaac Sim · ROS 2 · MoveIt 2 · OpenPI π0.5 · LeRobot · Fine-tuning