Reaction-Wheel Pendulum on a Momentum Budget

A cubli-style pendulum balanced only by the reaction torque of a saturating flywheel, so momentum is a finite budget and every controller is judged on wheel desaturation, not just staying up. Energy-pump swing-up succeeds 10/10, catching at 3.6 s. Both a 3-state discrete LQR and a 1.5M-step PPO balance 20/20 near upright, but a +0.8 rad/s poke pushes past the linear regime: the LQR loses all 20 while PPO saves 12 by pre-spinning the wheel through a larger fall. Documented honestly: the PPO policy cannot catch straight out of swing-up (a ~100 rad/s wheel it never trained on), so the handover stays with LQR.

OVERVIEW & MOTIVATION

The reaction-wheel pendulum is the fourth plant in a five-plant series on RL for underactuated control (cart-pole, Furuta pendulum, reaction-wheel pendulum, TWIP, ballbot), all built on the same sim2real recipe. It’s the purest test of momentum management: the only actuator is a motor-driven flywheel at the top of the pendulum arm, and Newton’s third law is the whole control problem — spin the wheel one way, the body torques the other way. Because the wheel speed is bounded, a controller that spends its whole budget correcting disturbances eventually runs out of authority and falls holding a “full” wheel. Every controller here is scored on wheel desaturation, not just time upright.

PHYSICAL SYSTEM & PARAMETERS

Modeled as a hobby build: a 3D-printed arm carrying a motor and a steel flywheel disc at the tip, pivoting freely at the base. State is [theta, theta_dot, omega] where theta is body angle from upright and omega is wheel speed relative to the body (what the motor encoder reports). Dynamics follow Spong’s reaction-wheel pendulum formulation with friction on both the pivot and the wheel bearing. Realism modeled in rwp_env.py: torque limit and deadband, wheel-speed saturation (torque that would push omega past the limit is zeroed, mimicking a saturated ESC), encoder quantization on body angle, hall-rate quantization on wheel speed, optional actuation delay and domain randomization, RK4 integration at 100 Hz. Energy conservation verified to 1e-9 with dissipation disabled.

PARAMETER	VALUE	UNIT
PENDULUM ARM MASS (m_rod)	0.05	KG
PIVOT-TO-WHEEL-AXIS LENGTH (L)	0.15	M
HUB (MOTOR+WHEEL) MASS (m_hub)	0.15	KG
FLYWHEEL DISC MASS (m_disc)	0.08	KG
FLYWHEEL RADIUS (r_disc)	0.04	M
GRAVITY (g)	9.81	M/S^2
PIVOT FRICTION (b_p)	1.5e-4	N·M·S/RAD
WHEEL BEARING FRICTION (b_w)	2.0e-5	N·M·S/RAD
MAX MOTOR TORQUE (TAU_MAX)	0.05	N·M
TORQUE DEADBAND	0.002	N·M
WHEEL SPEED LIMIT (OMEGA_MAX)	320 (~3000)	RAD/S (RPM)
CONTROL PERIOD (DT)	0.01 (100)	S (HZ)
RK4 SUBSTEPS PER STEP	5	—
BODY ENCODER RESOLUTION	600 x4 = 2400	COUNTS/REV
WHEEL SPEED QUANTIZATION	0.5	RAD/S PER STEP
EPISODE LENGTH	2000 (20)	STEPS (S)

METHOD

Observation space: Box([-pi, -30, -384], [pi, 30, 384]), i.e. [theta, theta_dot, omega] (omega bound is 1.2x OMEGA_MAX). Action space: Box(-1, 1), scaled to +/- TAU_MAX on the wheel motor. Reward: cos(theta) - 0.02*a^2 - 0.005*theta_dot^2 - 0.10*(omega/OMEGA_MAX)^2 — the last term is the wheel-speed regulation penalty that forces desaturation. Episodes terminate on fall (|theta| > 0.35 rad) outside swing-up mode, truncate at 2000 steps (20 s).

LQR: 3-state discrete design over [theta, theta_dot, omega], linearized numerically about upright and discretized at the 100 Hz control rate via cont2discrete + solve_discrete_are. Q = diag([100.0, 1.0, 2e-4]), R = [[50.0]]. Including omega in the state — not just [theta, theta_dot] — is what keeps the wheel desaturated; a 2-state design balances for a few seconds and then winds the wheel up to its limit.

PPO (train_ppo.py, Stable-Baselines3 MlpPolicy): 1,500,000 timesteps, 8 parallel envs, n_steps=512, batch_size=1024, learning_rate=3e-4, gamma=0.99, gae_lambda=0.95, ent_coef=0.005, seed 0. Training run takes roughly 10 minutes on CPU.

Swing-up: separate energy-pump controller, tau = KP * E * theta_dot, pumping the pendulum’s mechanical energy up to the upright equilibrium, then handing off to LQR for the catch.

RESULTS

Evaluated over 20 episodes of 20 s each, starting near upright:

METRIC	LQR (DISCRETE, 3-STATE)	PPO (1.5M STEPS)
BALANCE FROM NEAR-UPRIGHT	20/20	20/20
FINAL WHEEL SPEED (DESATURATION)	~1 RAD/S	~15 RAD/S
BODY POKE +0.8 RAD/S, PURE CONTROLLER	0/20	12/20
POKE WITH HYBRID RE-SWING-UP ASSIST	RECOVERED (1.8–2.4 S)	—

Swing-up succeeded 10/10 seeds tested, catch at 3.6 s, wheel speed peaking at 245 of 320 rad/s during the pump and bled back to ~1 rad/s by the 3-state LQR at handover.

Two findings stand out. First, the poke test separates the controllers: a +0.8 rad/s body disturbance is beyond the linear regime the LQR is designed around, so it loses all 20 trials, while PPO saves 12 by letting the body fall through a larger arc while pre-spinning the wheel — a genuinely nonlinear strategy the LQR can’t express. A hybrid architecture that re-enters swing-up whenever the body drops past 50 degrees recovers pokes with either controller. Second, distribution shift breaks the swing-up-to-balance handover for PPO: it balances 20/20 once near upright but can’t catch immediately after swing-up, when it inherits a wheel already spinning at 100+ rad/s — a state it never saw during training. Randomizing initial wheel speed during training (tried at +/-40% and +/-50% of OMEGA_MAX, up to 3M steps) made the balance task worse (9-15/20 survival) without fixing the catch, so the handover stays with LQR; curriculum or end-to-end training is flagged as the honest next step rather than claimed as solved.

Live interactive demo (live_demo.py) verified headless: swing-up in 3.66 s, pokes recovered in 1.77 s and 2.38 s.

USE CASES & APPLICATIONS

Reaction-wheel actuation without external contact is a real spacecraft attitude-control mechanism — momentum wheels and reaction wheels are the standard actuator for satellite pointing, and this plant is a tabletop analogue of that momentum-exchange problem, including the desaturation concern (spacecraft use magnetorquers or thrusters to dump momentum the same way this controller has to bleed wheel speed). More directly, this is exactly the actuation scheme behind Cubli-style self-balancing and self-righting cube robots, where a single flywheel per face provides all the control authority. Framed as applications the results motivate, not outcomes demonstrated here — no spacecraft or Cubli hardware was built or tested.

FILES & REPRODUCTION

rwp_env.py        custom Gymnasium env (all physics here)
lqr_baseline.py    3-state discrete LQR (wheel-speed feedback = desaturation)
swingup_ctrl.py    energy pump controller (importable, side-effect free)
swingup.py         swing-up experiment + animation
train_ppo.py        PPO training
evaluate.py         head-to-head evaluation + GIF
live_demo.py         real-time interactive demo (--headless N to verify)

pip install -r requirements.txt
python lqr_baseline.py    # 3-state discrete LQR: 20/20, wheel desaturated
python tune_swing.py      # swing-up gain study (10/10 across the grid)
python swingup.py         # swing-up + catch, saves the GIF
python train_ppo.py       # ~10 min CPU, saves ppo_balance.zip
python evaluate.py        # LQR-vs-PPO table + poke GIF
python live_demo.py       # interactive demo