Making robots faster and more successful with RL trained entirely in simulation

The problem: RL wasn't possible for them

A vertical robotics company had a working manipulation policy learned from teleop demonstrations. Like any imitation-learned policy, it inherited the hesitations and unneeded motion of the demonstrations it was trained on, and was least reliable on its hardest scenes. In production, throughput is what matters: how many successful tasks a robot ships per hour.

Reinforcement learning is the standard answer, but it was simply not an option for them before this. RL needs thousands of rollouts and dense feedback on which behaviors are better. This just isn’t practical nor scalable on a physical robot, which requires hours of manual environment resetting and operator time.

How the RL works

Fern’s world model is a learned simulator: it takes the robot’s camera views and a stream of joint actions, and predicts what the cameras see next. Because the customer’s policy runs closed-loop inside it just as it does on hardware, we can roll out and evaluate candidate behaviors at scale: thousands of rollouts in parallel, with no hardware in the loop.

Inside that simulator, we trained a small RL model that makes the policy faster and more reliable: trimming the wasted motion, and cleaning up behavior on the hard scenes where the base policy struggled.

• The method is policy-agnostic. It doesn’t require the customer’s policy weights, only access to the policy’s decisions. Their policy, training pipeline, and checkpoints stay untouched.
• All training signal came from the simulator. Every rollout and every reward signal, with no physical robot involved.
• Iteration runs at scale without hardware dependency. Rollouts are parallel and cheap, so the RL gets the volume of experience it needs, a throughput hardware can never provide.

The result on the real robots

The small RL model clearly improved the policy inside our world model. But the customer rightly wanted the only numbers that matter: does it hold up on the real robots? We deployed it on their hardware and evaluated the two head-to-head (base policy vs. base policy + Fern RL) as interleaved same-scene A/B pairs: 2,000 episodes across two robots over a week, every episode human-rated.

real-robot a/b · 2,000 scene-matched episodes · two robots, one week

With Fern’s RL model on top, the robots deliver 30% more successful completions per robot-hour, the operational bottom line, and the effect is highly statistically significant (p < 0.001). The gain comes from both directions: completions got faster, and task success rate improved 7% in absolute terms (p = 0.0008), with the largest gains on exactly the scenes where the base policy struggled most. Faster, and more reliable.

The improvement the model showed inside the world model is the improvement the customer measured on their hardware: no sim-to-real gap. That’s the payoff of training against a simulator that is itself learned from the robot’s real data: the behaviors that win in sim are winning in the real world.

The flywheel

This first result set up something more valuable than a one-off speedup: a continuous improvement loop that is now running with this customer.

1. 1

   Real robots collect scenes at scale

   The customer’s fleet runs production rollouts as usual. Every initial scene it encounters, including the rare, awkward ones, becomes a starting point for simulation.

2. 2

   Every scene is rolled out in Fern's world model

   Their policy plus the current RL model runs closed-loop in sim from each of those real starting scenes: thousands of rollouts, no hardware in the loop.

3. 3

   The rollouts are reviewed in sim

   New failure modes and remaining slow behavior are surfaced from the simulated rollouts, at video-review cost and with zero robot time.

4. 4

   An improved RL model ships

   Those findings train the next model, fixing the newly-found failures and pushing speed further. It deploys on top of the unchanged base policy, and the loop repeats.

Each turn of the loop, the robot gets faster and more reliable as the fleet continuously runs in the production environment.

Why this matters

• This makes RL possible for robotics. On hardware, RL is serial: one robot, one rollout at a time, with environment resets between every attempt and environment/hardware inconsistencies polluting the reward signal. In the world model, rollouts run in parallel at whatever scale the training needs. Every rollout and every training step happened in sim. For this customer, RL went from impossible to routine.
• No sim-to-real gap, and we proved it. The improvement Fern’s RL showed in sim is the improvement the customer measured when we deployed it on their real robots.
• It improves the thing robotics companies care about most: the policy. The robots now deliver 30% more successful completions per hour, with a success rate 7% higher, with no change to the customer’s underlying policy or infrastructure, and a flywheel that keeps improving it.