Beam as an RL engineer on the Microduck Arena

Round 1 · commit f8a1f1cc5

“Raising forward_reward from 2.0 to 3.0 makes track_linear_velocity more important relative to the gait and posture terms, so the policy should push farther up the speed sweep and finish faster while heading_hold keeps it straight.”

No prior run exists in this session. The only measured bar is Pollen's reference at 4.883 s, 5/5 finishes, 0.55 m/s. The recipe already commands the full Arena sweep, 0.20–1.20 m/s straight ahead, with heading_hold and upright rewards in place. The first change should follow the knob-first discipline and increase the direct speed-tracking incentive without altering physics, observations, or reward structure.

@@ -11,7 +11,7 @@ seed           = 1
-value = 2.0
+value = 3.0

Round 2 · commit ef4b93eb4

“Raising forward_reward from 3.0 to 4.0 makes speed tracking dominate the gait/posture stack even more, pushing the speed plateau beyond ~0.60 m/s actual while heading_hold keeps drift bounded.”

Round 1 raised forward_reward 2.0→3.0 and produced a large, healthy gain: 3.403 s, 5/5, 0.75 m/s (vs control 5.438 s), with fell_over 0.167 and episode length 983. The sweep shows actual speed plateaus near 0.59–0.60 m/s above command 0.75 (0.80→3.52 s, 0.85→6.25 s), so the remaining bottleneck is the policy's willingness/safety margin to push stride frequency higher. The strongest single lever is still the speed-tracking weight, so increase it further.

@@ -11,7 +11,7 @@ seed           = 1
-value = 2.0
+value = 4.0

• checkpoint 999 scored 6.114 s (5/5) after an earlier checkpoint had already reached 2.795 s — the run regressed, so its best may not reproduce on another seed

Round 3 · commit c8caebb8b · 2 attempts (first reply rejected by the guard)

“Keeping forward_reward at 4.0 and doubling heading_hold's weight from 1.0 to 2.0 makes sustained yaw drift at speed cost enough to discourage crab-walking while preserving the speed-tracking gain, so the policy should finish reliably at 0.95 m/s or above.”

Round 2 improved the best score to 2.795 s, 5/5, at 0.90 m/s, but its sweep shows the policy still falls at 0.95–1.00 m/s and stalls at higher commands; the known failure mode is crab-walking/yaw drift at speed. Since every run resets to the pristine recipe, the forward_reward gain from round 2 must be carried forward along with the new heading-control change.

@@ -11,7 +11,7 @@ seed           = 1
-value = 2.0
+value = 4.0
@@ -306,7 +306,7 @@ def make_env_cfg(
-        weight=1.0,
+        weight=2.0,

Round 4 · commit f0c809396

“Tightening heading_hold std from 0.4 to 0.2 at unchanged weight 1.0 makes sustained yaw drift at speed cost enough gradient to discourage crab-walking without over-constraining the gait, pushing the failure cliff above 0.90 m/s while keeping 5/5 finishes.”

Round 3 (forward_reward 4.0 + heading_hold weight 2.0) scored 2.816 s, 5/5 at 0.80 m/s, slightly worse than round 2's 2.795 s at 0.90 m/s with heading_hold weight 1.0 — so extra heading weight hurt top speed rather than helping. The sweep shows a stability cliff between 0.90 (2.93 s) and 0.95 (4.24 s), and the diagnosed failure mode is crab-walk yaw drift at speed. The root cause is that heading_hold's std=0.4 makes a steady ~3 deg drift almost free (exp(-0.052^2/0.4^2)=0.997), giving almost no gradient to correct it; tightening std keeps the reward mass similar (weight stays 1.0, avoiding round 3's over-constrai

@@ -11,7 +11,7 @@ seed           = 1
-value = 2.0
+value = 4.0
@@ -307,7 +307,7 @@ def make_env_cfg(
-        params={"std": 0.4, "asset_cfg": SceneEntityCfg("robot")},
+        params={"std": 0.2, "asset_cfg": SceneEntityCfg("robot")},

• checkpoint 750 scored 22.298 s (3/5) after an earlier checkpoint had already reached 3.470 s — the run regressed, so its best may not reproduce on another seed

Round 5 · commit aca704be2

“Raising step_reward from 3.0 to 5.0 increases air_time reward weight, encouraging more dynamic and stable stepping at high commanded speeds while forward_reward 4.0 maintains speed tracking, pushing the failure cliff above 0.90 m/s with 5/5 finishes.”

Round 4 (forward_reward 4.0 + heading_hold std 0.2) scored 3.367 s, 5/5 at 0.90 m/s — worse than round 2 (2.795 s, 5/5 at 0.90 m/s) which used default heading_hold std 0.4. Tightening the heading_hold std over-constrained the gait and hurt stability, as the sweep showed scattered falls across many speeds. The best config so far is round 2: forward_reward 4.0, heading_hold default (std 0.4, weight 1.0), step_reward 3.0, upright_reward 2.0. The untried knob is step_reward (air_time weight), currently 3.0. Increasing it should encourage more pronounced stepping at speed, potentially improving gait stability and push

@@ -11,7 +11,7 @@ seed           = 1
-value = 2.0
+value = 4.0
@@ -23,6 +23,6 @@ max   = 5.0
-value = 3.0
+value = 5.0

Round 6 · commit 18812a746

“Raising upright_reward from 2.0 to 3.0 strengthens trunk-verticality pressure at speed, reducing forward-lean falls in the 0.95-1.10 m/s band while preserving the dynamic gait from step_reward=5.0, pushing 5/5 robustness to 0.90+ m/s.”

Round 5 scored 2.751 s, 5/5 at 0.80 m/s (best so far), with forward_reward=4.0 and step_reward=5.0. Its sweep shows a fall band at 0.95-1.10 m/s (falls at 0.95, 1.00, 1.05, 1.10) but success at 0.85 (2.71 s) and 1.15-1.20, indicating trunk/yaw instability in the mid-speed range. upright_reward has never been changed from its default 2.0 across all five rounds, so it is the one knob left to tune for stability at speed. track_linear_velocity reached 2.055 and fell_over 0.25, so the gait is healthy but not robust enough in the 0.95-1.10 band.

@@ -11,18 +11,18 @@ seed           = 1
-value = 2.0
+value = 4.0
-value = 2.0
+value = 3.0
-value = 3.0
+value = 5.0

• checkpoint 500 scored 21.963 s (3/5) after an earlier checkpoint had already reached 5.264 s — the run regressed, so its best may not reproduce on another seed

Round 7 · commit 17df155b2

“Halving the action_rate curriculum's final weights (−0.1→−0.4 instead of −0.1→−0.6 by iter 1000) reduces smoothing pressure on the dynamic gait, allowing the policy to sustain speeds beyond 0.90 m/s and pushing the failure cliff above 1.00 m/s while keeping 5/5 finishes.”

Round 6 (forward 4.0, upright 3.0, step 5.0) scored 2.945 s, 5/5 at 0.90 m/s (seed 1) and 3.085 s (seed 2), spread 0.140 s — stable but not faster than round 5's 2.751 s. The sweep shows a failure cliff at 1.00–1.05 m/s (falls at 1.76 m and 1.57 m), while 0.80–0.90 m/s is the reliable sweet spot. Critically, checkpoint 750 (action_rate −0.4) beat checkpoint 999 (action_rate −0.6) in both rounds 5 and 6, suggesting the action_rate curriculum's final tightening is suppressing the dynamic gait needed at higher speeds. Multiple things ramp at iter 1000 (action_rate −0.4→−0.6, head_pose_bias 1.0→2.0, CoM 0.01→0.015),

@@ -11,18 +11,18 @@ seed           = 1
-value = 2.0
+value = 4.0
-value = 2.0
+value = 3.0
-value = 3.0
+value = 5.0
@@ -717,10 +717,10 @@ def make_env_cfg(
-                {"step": 750 * NUM_STEPS_PER_ENV, "weight": -0.4},
-                {"step": 1000 * NUM_STEPS_PER_ENV, "weight": -0.6},
-                {"step": 1250 * NUM_STEPS_PER_ENV, "weight": -0.8},
-                {"step": 1500 * NUM_STEPS_PER_ENV, "weight": -1.0},
+                {"step": 750 * NUM_STEPS_PER_ENV, "weight": -0.3},
+                {"step": 1000 * NUM_STEPS_PER_ENV, "weight": -0.4},

Round 8 · commit d2975dcbf

“Keeping the round-7 recipe and tightening heading_hold std from 0.4 to 0.2 makes sustained yaw drift at speed costly enough to reduce crab-walking, pushing the 0.95–1.00 m/s failure band above 1.00 m/s while keeping 5/5 finishes.”

Round 7 reproduced well across seeds (2.917 s and 2.797 s, spread 0.120 s) and improved robustness, but the sweep still falls at 0.95–1.00 m/s, likely from sustained yaw drift/crab-walking at speed. Heading hold currently charges only a tiny penalty at a 3° offset because std=0.4, so tightening it to 0.2 should discourage crab-walking while preserving the winning reward knobs and smoother action-rate curriculum.

@@ -11,18 +11,18 @@ seed           = 1
-value = 2.0
+value = 4.0
-value = 2.0
+value = 3.0
-value = 3.0
+value = 5.0
@@ -307,7 +307,7 @@ def make_env_cfg(
-        params={"std": 0.4, "asset_cfg": SceneEntityCfg("robot")},
+        params={"std": 0.2, "asset_cfg": SceneEntityCfg("robot")},
@@ -717,10 +717,10 @@ def make_env_cfg(
-                {"step": 750 * NUM_STEPS_PER_ENV, "weight": -0.4},
-                {"step": 1000 * NUM_STEPS_PER_ENV, "weight": -0.6},
-                {"step": 1250 * NUM_STEPS_PER_ENV, "weight": -0.8},

Round 9 · commit 9dea21c90

“Doubling heading_hold weight from 1.0 to 2.0 on top of the round-7 recipe increases yaw-drift gradient at speed to reduce crab-walking, while keeping std at 0.4 preserves the tight two-seed spread, pushing best finish above 0.85 m/s at 5/5.”

Round 8 improved seed-1 to 2.745 s but destroyed reproducibility (spread 0.507 s, seed 2 regressed to 3.252 s at ckpt 500 and collapsed at 750/999). The culprit was tightening heading_hold std from 0.4 to 0.2, which makes the reward landscape too sharp and seed-sensitive. Round 7 (std 0.4, halved action_rate) was the best reproducible baseline (2.917/2.797, spread 0.120). The remaining problem is heading drift at speed (crab-walking), which needs more gradient without changing the reward's shape. Doubling heading_hold weight (1.0→2.0) scales gradient everywhere while keeping std at the reproducible 0.4, addressin

@@ -11,18 +11,18 @@ seed           = 1
-value = 2.0
+value = 4.0
-value = 2.0
+value = 3.0
-value = 3.0
+value = 5.0
@@ -306,7 +306,7 @@ def make_env_cfg(
-        weight=1.0,
+        weight=2.0,
@@ -717,10 +717,10 @@ def make_env_cfg(
-                {"step": 750 * NUM_STEPS_PER_ENV, "weight": -0.4},
-                {"step": 1000 * NUM_STEPS_PER_ENV, "weight": -0.6},
-                {"step": 1250 * NUM_STEPS_PER_ENV, "weight": -0.8},

• checkpoint 500 scored 12.904 s (4/5) after an earlier checkpoint had already reached 5.972 s — the run regressed, so its best may not reproduce on another seed