> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bfl.ml/llms.txt
> Use this file to discover all available pages before exploring further.
# SO-101 with LeRobot
> Train a task LoRA, preserve the checkpoint’s camera and control conventions, and evaluate it on your SO-101.
Adapt the prepared SO-101 checkpoint with LeRobot. LoRA updates the trunk while
the action and conditioning heads train in full. Keep the base checkpoint and
its saved processors available for training, resume, and inference.
The clips below show real robot demonstrations. Playback is accelerated and
pauses between plans are removed; they are not latency benchmarks or validation
of every later integration revision.
"Put the blue box into the container." Trained with about 200 teleoperated demonstrations. Top camera, 4× speed, pauses between plans cut.
"Put the white box from the green cup to the container."
"Put the screwdriver into the container."
"Put the tissues into the container."
These objects were not included in the task-specific pick demonstrations.
## Requirements
* Linux, Python 3.12, and an NVIDIA GPU with a CUDA 12.8-compatible driver.
* Access to the model weights and the LeRobot integration repository below.
* A calibrated SO-101 and two cameras for live rollout.
* Your own LeRobot v3 dataset at **30 Hz**, with six absolute commanded actions,
six measured-state values, and task instructions.
Joint order is shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll,
and gripper. The delivered standalone SO-101 profile uses degrees for the five
arm joints and percentage points for the gripper. Match the LeRobot package's
saved processors, joint order, units, and calibration; do not normalize twice.
The SO-101 recipe uses **30 Hz**; DROID and the games use 15 Hz. The pinned
checkpoint predicts 42 actions and executes the first 32, about 1.07 seconds
at 30 Hz, before replanning. LeRobot executes the number of actions specified
by the downloaded package's `n_action_steps`.
## Install LeRobot
Use a separate environment from standalone `flux-action`. FLUX 3 support ships
in LeRobot main, which the released packages target:
```sh theme={null}
git clone https://github.com/huggingface/lerobot.git
cd lerobot
uv venv --python 3.12
source .venv/bin/activate
uv pip install 'torch==2.11.0' 'torchvision==0.26.0' \
--index-url https://download.pytorch.org/whl/cu128
uv pip install -e '.[training,flux3,peft,diffusion]'
uv pip install 'natten==0.21.6+torch2110cu128' --find-links https://whl.natten.org/
ffmpeg -version
```
Install FFmpeg with video decoding support if the last command is unavailable.
The NATTEN wheel above is for PyTorch 2.11/CUDA 12.8; choose a matching wheel if
you change that pair. Run subsequent commands from this checkout with its
virtual environment active.
## Download the base
Use the released packages from the
[FLUX 3 Action collection](https://huggingface.co/collections/black-forest-labs/flux-3-action)
rather than the integration's older temporary download helper. Download the
SO-101 policy, then the shared encoders at the base revision its config references:
```sh theme={null}
hf auth login
hf download black-forest-labs/flux-3-action-so101 \
--revision c9e13b2aca6a0a472b3ea03fd90cff32d3e85849 \
--local-dir models/flux-3-action-so101
hf download black-forest-labs/flux-3-action-base \
--revision 62878e2925e59b7a89ec14463ce89932624c490d \
--include 'video_vae.safetensors' --include 'text_encoder/*' \
--local-dir models/flux-3-action-base
policy_dir="$(pwd)/models/flux-3-action-so101"
encoders_dir="$(pwd)/models/flux-3-action-base"
```
The SO-101 directory contains the policy, saved processors, and normalization
statistics. Keep both directories available for training, resume, and adapter
inference. Supply your own demonstrations; these downloads do not include them.
Inspect the package's settings:
```python theme={null}
import json
from pathlib import Path
root = Path("models/flux-3-action-so101")
config = json.loads((root / "config.json").read_text())
for key in ("camera_keys", "camera_layout", "fps", "chunk_size", "n_action_steps"):
print(key, config.get(key))
assert config["fps"] == 30
```
Train with the integration's `examples/flux3/lora.json`. Keep the package's
normalization statistics fixed for the adapter. Different statistics require a separately
prepared base and matching processors, not a substitution during rollout.
## Camera keys
| Physical camera | Checkpoint key | Position in the model image |
| -------------------- | -------------------------- | --------------------------- |
| Fixed scene camera | `observation.images.scene` | Left |
| Wrist-mounted camera | `observation.images.wrist` | Right |
Each view is resized to 256 × 256 and joined into a **512 × 256** image
(width × height). Supply separate streams. Old instructions using `top` and
`front` as checkpoint keys do not describe this package.
If your dataset calls the scene camera `top` and the wrist camera `gripper_cam`,
append this argument to the training command:
```sh theme={null}
--rename_map='{"observation.images.top":"observation.images.scene","observation.images.gripper_cam":"observation.images.wrist"}'
```
Use the same physical mapping at rollout. A rename map changes keys; it cannot
identify a swapped camera. Inspect the composed image before VAE encoding:
scene on the left, wrist on the right.
## Train
The shared preset trains on one GPU with batch 2 and accumulation 4:
```sh theme={null}
python -m lerobot.scripts.lerobot_train \
--config_path=examples/flux3/lora.json \
--policy.path="$policy_dir" --policy.device=cuda \
--policy.video_vae_id="$encoders_dir/video_vae.safetensors" \
--policy.text_encoder_id="$encoders_dir/text_encoder" \
--dataset.repo_id=YOUR_ORG/YOUR_SO101_DATASET \
--output_dir=outputs/so101_lora
```
Replace the dataset ID and add the camera rename map only if needed. Confirm
the loaded settings before starting: rank/alpha 32, adapter/head learning rates
`1e-4`/`5e-4`, BF16, gradient checkpointing, and EMA decay `0.999`.
New adapters are created through the top-level `peft` config. The flag
`--policy.use_peft=true` is for loading an existing adapter.
The preset uses effective batch 8 and `steps=10000`. LeRobot counts `steps`,
`save_freq`, and `eval_steps` in **microsteps**: with accumulation 4, that is
2,500 optimizer updates, and 500 microsteps equal 125 updates. Increase the
budget or GPU count explicitly for your dataset, and check peak GPU memory on
your hardware.
### Resume and select weights
```sh theme={null}
lerobot-train \
--config_path=outputs/so101_lora/checkpoints/last/pretrained_model/train_config.json \
--resume=true
```
Resume from the raw checkpoint to restore optimizer and EMA state. For
inference, compare `pretrained_model` with its `pretrained_model_ema` sibling
using the same tasks, starts, and control settings. EMA helps some runs but is
not automatically the better SO-101 checkpoint. Retain the adapter's original
base and encoder files.
## Roll out
Use the integration's synchronous runner for a trained adapter. Set the port,
calibrated robot ID, camera devices, and instruction for your setup:
```sh theme={null}
lerobot-rollout \
--strategy.type=base --inference.type=sync \
--policy.path=outputs/so101_lora/checkpoints/last/pretrained_model \
--policy.use_peft=true --policy.device=cuda \
--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101 \
--robot.use_degrees=true \
--robot.cameras='{"scene":{"type":"opencv","index_or_path":0,"width":640,"height":480,"fps":30},"wrist":{"type":"opencv","index_or_path":1,"width":640,"height":480,"fps":30}}' \
--fps=30 --task="put the blue box into the container" --duration=30
```
This executes commands on the arm. Use the robot ID and calibration that match
your training data, and confirm the two live camera views before rollout.
The base strategy runs without recording or uploading episodes. Swap the
checkpoint path to compare raw and EMA adapters.
The runner applies the saved processors and resets state between episodes.
This policy supports synchronous rollout. For direct Python calls and processor
handling, see the [integration guide](https://huggingface.co/docs/lerobot/main/en/flux3#inference).
The clips show earlier real-robot runs. Evaluate task success and latency on
your own arm.
For full fine-tuning instead of a task LoRA, follow the separate
[standalone SO-101 workflow](https://github.com/black-forest-labs/flux-action/blob/main/docs/so101-lora.md#optional-full-fine-tuning).