"Fly under the coffee table and hover." Tested on a room layout and instruction wording absent from the fine-tuning data.
## Simulation | | | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | World | A living room in Isaac Sim: sofa, coffee table, bookcase, window, landing pad, clutter. Sofa and bookcase against a random wall or free-standing, pad and clutter moved, the drone facing a random direction at take-off. Layout changes per seed | | Onboard camera | 256×256, altitude and heading drawn onto the frame | | Action | `[forward, lateral, up, yaw]` in `[-1, 1]`, velocity commands with first-order lag, 15 Hz | | Instructions | Instructions naming destinations: "take off, fly under the table, and hover", "take off, fly to the sofa, and hover", "take off, fly to the pad, and land" | | Expert | Scripted pilot with a map: take off, yaw to bearing, fly waypoints from a visibility graph, hover, land | ## What changed in the recipe The `rotor` dataset adapter reads an instruction per episode and prepends `fly the drone: `. Use the same prefix at inference. The recipe uses four action dimensions, a 512 × 512 canvas, and 3,000 updates. Start with the [local game config](/flux_3/flux3_action_finetuning#configure-training), then create a drone-only config: ```python theme={null} import json from pathlib import Path config = json.loads(Path("configs/games/local.json").read_text()) config["source_root"] = "rotor=outputs/data/rotor" config["output_dir"] = "outputs/drone" Path("configs/games/drone-local.json").write_text(json.dumps(config, indent=2)) ``` The remaining game settings, including `caption_dropout=0.0`, stay unchanged. The directory `outputs/data/rotor` needs an index such as: ```json theme={null} { "length": 300, "action_dim": 4, "episodes": [ {"file": "ep_0000.npz", "task": "take off, fly under the table, and hover"}, {"file": "ep_0001.npz", "task": "take off, fly to the pad, and land"} ] } ``` Each NPZ contains uint8 RGB `frames` with shape `(300, 256, 256, 3)` and float32 `action` with shape `(300, 4)`. The unprefixed `task` records the instruction followed by the pilot. The adapter prepends `fly the drone: ` once. Two entries illustrate the format; supply enough episodes for the training batch topology. Use the fine-tuning guide's data checks and short validation run with the new config, then train and export: ```sh theme={null} uv run torchrun --nproc_per_node 8 -m flux_action.cli train \ --config configs/games/drone-local.json uv run flux-action export-checkpoint --checkpoint outputs/drone/step-3000 \ --output outputs/drone-export --profile ema_0p10 --dtype bfloat16 ``` This launch uses the current eight-GPU configuration. It is not evidence of the original drone experiment's hardware or wall time, which the notes do not specify. 800 episodes of 20 seconds at 15 Hz, recorded from the scripted pilot. Five scenarios: standard start, random start, already airborne, near an obstacle, and mid-flight disturbances. ## Held-out layouts, unseen wording Evaluation uses ten layouts excluded from fine-tuning and reworded instructions. For example, "take off, fly to the bookcase, and hover" becomes "go over to the bookshelf and wait there". The simulator waits for each plan across 89 evaluation flights: | Evaluation instruction | Fine-tuning instruction | Completed | | ----------------------------------------------------- | ----------------------------------------- | --------- | | fly to the window and hover in front of it | take off, fly to the window, and hover | 10 / 10 | | take off and land on the pad | take off, fly to the pad, and land | 9 / 10 | | fly under the coffee table and hover | take off, fly under the table, and hover | 8 / 10 | | go over to the bookshelf and wait there | take off, fly to the bookcase, and hover | 7 / 10 | | fly to the bookcase and hover in front of it | take off, fly to the bookcase, and hover | 6 / 10 | | fly around the coffee table and hover on the far side | take off, fly behind the table, and hover | 5 / 10 | | go to the couch | take off, fly to the sofa, and hover | 5 / 10 | | fly to the sofa and hover in front of it | take off, fly to the sofa, and hover | 3 / 9 | | fly to the coffee table and hover above it | take off, fly to the table, and hover | 2 / 10 | The model completed 55 of 89 flights, with 2 crashes. Success was highest for the window, landing pad, and under-table tasks, and lowest for the sofa and above-table tasks. "Bookshelf" and "couch" were absent from the fine-tuning instructions."Fly to the window and hover in front of it." Completed.
"Go over to the bookshelf and wait there." Completed; "bookshelf" was absent from the fine-tuning instructions.
"Go to the couch." Completed.
"Fly to the coffee table and hover above it." One of two successful flights in ten attempts.
"Fly to the bookcase and hover in front of it." Failed to reach the bookcase.