Furuta Pendulum, End-to-End PPO Swing-Up

A rotary inverted pendulum with full 2-DOF mass-matrix dynamics, reflected gearbox inertia, motor deadband, dual encoder quantization, and a bouncy cable-wrap hard stop. The comparison is the point: a feedback-linearized energy pump, tuned over ten seeds, caught only 7/10 with a 2.4 s average, while an end-to-end PPO policy swings up 20/20 and reaches upright in 0.71 s by whipping the arm so the pole rises centrifugally. Both a discrete LQR and PPO balance 20/20 near upright, but the continuous-time LQR and a balance-only PPO catch both failed (0/20 and 0/10) — distribution-shift failures kept in the repo as the interesting part.

OVERVIEW & MOTIVATION

The Furuta pendulum is the classic underactuated-control benchmark: a motor drives a horizontal arm, a free pendulum hangs off the arm’s tip, and only the arm is actuated. Balancing near upright is a solved problem (LQR does it). Swinging up from hanging and catching the pole is the hard part, and this project runs the honest version of that fight: five rounds of hand tuning on a classical energy-pump swing-up, capped at 7/10 catches, versus one end-to-end PPO policy that gets 20/20 and reaches upright three times faster. The env models a hobby-scale rig (motor torque limit and deadband, reflected gearbox inertia, dual quadrature encoders, a bouncy cable-wrap limit) rather than an idealized double pendulum, so the failures — and the fix — are the kind sim2real work actually runs into.

PHYSICAL SYSTEM & PARAMETERS

2-DOF Furuta dynamics (arm and pendulum both treated as rods) solved exactly in mass-matrix form each RK4 substep; energy conservation verified to 1e-9 with torque and friction disabled. The arm’s reflected gearbox rotor inertia (J_m, a 25:1 gearbox on a ~1.6e-6 kg·m² rotor) dominates the bare arm rod inertia — without it the pole out-muscles the arm and flings it around, so this term is what makes the arm feel “stiff” the way a real Furuta rig does.

PARAMETER	VALUE	UNIT
Arm length (pivot to pendulum joint), L1	0.10	m
Arm mass, m1	0.06	kg
Pendulum mass, m2	0.05	kg
Pendulum pivot-to-CoM, l2	0.10	m
Reflected rotor inertia, J_m	1.0e-3	kg·m²
Arm viscous friction, b_arm	8e-4	N·m·s/rad
Pendulum viscous friction, b_pole	5e-5	N·m·s/rad
Max motor torque, TAU_MAX	0.15	N·m
Motor deadband	0.006	N·m
Control period (50 Hz)	0.02	s
Physics substeps per control step	10	—
Encoder resolution (both joints)	600 P/R ×4	counts
Cable-wrap hard stop	±2.5π	rad
Hard-stop restitution	0.3	—
Episode length	1000 (20)	steps (s)
Fall threshold (balance task), |θ|	0.35 (~20°)	rad

TAU_MAX is roughly 3x the static gravity torque on the pendulum (m2·g·l2), comparable authority to a Quanser Qube.

METHOD

Observation: [phi, phi_dot, theta, theta_dot], both angles quantized to the 600 P/R ×4 encoder resolution, theta wrapped to (-π, π]. The end-to-end swing-up policy uses a trig-wrapped observation instead — [phi, phi_dot, sin(theta), cos(theta), theta_dot] — since raw theta’s wrap discontinuity sits exactly where a swing-up policy spends its time and cripples learning if fed raw.

Action: Box(-1, 1), scaled to ±TAU_MAX N·m, zeroed below the deadband.

Reward (balance): cos(theta) - 0.05*(phi/pi)^2 - 0.02*a^2 - 0.005*thetadot^2 - 0.0005*phidot^2. Swing-up adds: an energy-shaped term, 1 - min(|E|/(2*mgl), 1.5) where E is pendulum swing energy (0 at upright rest, -2mgl hanging), plus a -2.0 penalty on hitting the cable-wrap stop.

LQR: numeric central-difference linearization about upright, continuous A/B converted to discrete (Ad, Bd) at the 50 Hz control rate via cont2discrete, then a discrete algebraic Riccati equation (solve_discrete_are) for gain K. Q = diag([1.0, 0.05, 100.0, 1.0]) on (phi, phi_dot, theta, theta_dot), R = 20.0. A continuous-time design was tried first and is unstable here — one 20 ms step of saturated torque changes arm speed by ~4 rad/s, so sampling can’t be neglected.

PPO (Stable-Baselines3, MlpPolicy):

RUN	STEPS	ENVS	n_steps	batch_size	lr	gamma	gae_lambda	ent_coef
Balance	1,500,000	8	512	1024	3e-4	0.99	0.95	0.005
End-to-end swing-up	4,000,000	8	1024	2048	3e-4	0.995	0.95	0.01

RESULTS

TASK	LQR (DISCRETE)	PPO
Balance from near-upright, 20 eps	20/20	20/20
Pole poke +1.2 rad/s	20/20	20/20
Swing-up: energy ctrl + LQR catch	7/10, 2.4 s avg	—
Swing-up: energy ctrl + PPO catch	—	0/10
Swing-up: end-to-end PPO	—	20/20, 0.71 s

Continuous-time LQR (untuned for sampling): 0-15/20, discarded in favor of the discrete design above. First end-to-end PPO attempt: 0/20, episodes dying at 27 steps because early exploration slammed the cable-wrap termination in ~0.5 s before anything could be learned — fixed by making the wrap limit a bouncy stop plus the energy-shaped reward term, after which the same setup reached 20/20. Interactive live-demo run (headless, deterministic pole-poke): swing-up 4.2 s, pokes 1.8 s and 1.4 s.

USE CASES & APPLICATIONS

The rotary inverted pendulum is a standard bench for underactuated-control education (Quanser Qube and similar teaching rigs use the same geometry) and for testing direct-drive/geared actuator control under torque limits and deadband. The failure catalogue here — continuous-time LQR at too coarse a sample rate, energy pumps stalling in a centrifugal spin trap, a balance-only policy failing under distribution shift when the arm is spinning — maps directly onto the kinds of problems that show up moving a controller from a smooth benchmark model to hardware with encoder quantization, actuator deadband, and hard travel limits. The reflected gearbox inertia term is the same effect that governs how “stiff” any geared, direct-drive-adjacent joint feels against an external reaction torque, relevant to underactuated legged and manipulator joints generally.

FILES & REPRODUCTION

• furuta_env.py — Gymnasium env, all physics (mass-matrix dynamics, RK4, deadband, quantization, cable-wrap stop)
• lqr_baseline.py — discrete LQR design + 20-episode eval
• swingup_ctrl.py — feedback-linearized energy pump (best classical result: 7/10)
• swingup.py — hybrid swing-up experiment + GIF
• train_ppo.py — PPO balance training
• train_ppo_swingup.py — end-to-end PPO swing-up (trig-obs wrapper)
• evaluate.py — full comparison table + GIFs
• live_demo.py — real-time interactive demo (--headless N to verify)
• tune_lqr.py, tune_swing*.py, debug_*.py — tuning/diagnosis history

pip install -r requirements.txt
python lqr_baseline.py         # discrete LQR: 20/20
python swingup.py              # energy swing-up + LQR catch + GIF
python train_ppo.py            # PPO balance (~10 min CPU)
python train_ppo_swingup.py    # end-to-end PPO swing-up (~30 min CPU)
python evaluate.py             # all comparisons + e2e GIF
python live_demo.py            # interactive demo