Cart-Pole Swing-Up & Balance, Sim-to-Real Ready

Flagship of a sim-to-real series: a belt-driven cart-pole in a custom Gymnasium environment with motor deadband, 600 P/R encoder quantization, cart and pivot friction, and solid end walls that transfer impulse into the pole. A PPO policy trained over 1.5M steps balances 100% across 20 episodes and recovers from mid-run pokes (≤3.5°, under 1 s), matching the discrete LQR baseline that also balances 100%. Swing-up from hanging uses an Åström-style energy pump gated to the ±0.4 m rail, handing over to PPO or LQR at a 6.4 s catch. The same code targets an Arduino-based physical rig over a 50 Hz serial link.

OVERVIEW & MOTIVATION

Phase 1 of a sim2real program: build a cart-pole simulation realistic enough that a policy trained inside it stands a chance on real hardware, then answer the question every RL-vs-classical argument dodges — on a plant this messy, does a learned policy actually buy anything over a hand-derived LQR gain? The env is deliberately not the frictionless, bang-bang CartPole-v1 toybox: it carries motor deadband, encoder quantization, viscous friction, and solid end walls that whip the pole on impact. Both controllers are evaluated on identical rollouts so the comparison is head-to-head, not cherry-picked.

PHYSICAL SYSTEM & PARAMETERS

State is [x, x_dot, theta, theta_dot] with theta = 0 at upright; action is a continuous force command in [-1, 1] scaled to ±F_MAX. Dynamics integrate with RK4 at 10 substeps per 50 Hz control step — the rate a PC↔Arduino serial link actually sustains.

PARAMETER	VALUE	UNIT
CART MASS (M)	0.60	kg
POLE MASS (m)	0.15	kg
PIVOT-TO-COM LENGTH (l)	0.25	m
GRAVITY (g)	9.81	m/s^2
CART FRICTION (b_cart)	1.5	N·s/m
PIVOT FRICTION (b_pole)	0.002	N·m·s/rad
MAX MOTOR FORCE (F_MAX)	8.0	N
DEADBAND	0.30	N
RAIL HALF-LENGTH	±0.40	m
WALL RESTITUTION	0.35	—
CONTROL PERIOD (DT)	0.02 (50 Hz)	s
PHYSICS SUBSTEPS	10	per step
PENDULUM ENCODER	2400 (600 P/R × 4)	counts/rev
CART QUANTIZATION	0.0005	m/count
FALL TERMINATION (balance mode)	±0.35 (~20)	rad (deg)
EPISODE LENGTH	1000 (20)	steps (s)

METHOD

Environment. Observation is the quantized 4-vector above (theta wrapped to ±π); action is a scalar force with motor deadband applied before the RK4 integration. Reward is cos(theta) - 0.15*(x/RAIL)^2 - 0.02*a^2 - 0.005*theta_dot^2, with a −2.0 penalty on any wall strike. End walls conserve the pole’s angular momentum about the pivot on collision (dtheta_dot = -(3/4l)·cos(theta)·dx_dot), so a rail slam visibly kicks the pole rather than just stopping the cart.

LQR. Dynamics are linearized about upright via finite-difference Jacobians of the same _deriv function PPO trains against (lqr_baseline.py), then solved as a continuous-time algebraic Riccati equation with Q = diag([8, 1, 40, 2]), R = 0.05, giving state-feedback gain K. The controller sees the same quantized observations as PPO.

PPO. Stable-Baselines3 PPO, MlpPolicy with a [64, 64] net, trained via n_steps=512, batch_size=1024, learning_rate=3e-4, gamma=0.99, gae_lambda=0.95, ent_coef=0.005, across 8 parallel envs for 1.5M total timesteps (~6 min on CPU). The README notes the same architecture solves toy CartPole-v1 in 50k steps; the 30x step increase here is attributed to the deadband and encoder quantization.

Swing-up. An Åström-Furuta energy-pump controller (swingup_ctrl.py, gain K_E = 14.0, cart-recentering gain K_X = 1.2) drives the pole from hanging, gated to only pump while it pushes the cart toward center given the short ±0.4 m rail. Control hands off to PPO or LQR once the pole enters a ±25° cone with |theta_dot| < 2.5 rad/s, and hands back to swing-up if it’s lost past ±50°.

RESULTS

Task	LQR	PPO (1.5M steps)
Balance from near-upright, 20 episodes	100%	100%
Recovery from mid-run pole poke	yes	yes (≤3.5°, <1 s)
Swing-up from hanging + catch	caught at t = 6.4 s	caught at t = 6.4 s

Both controllers balance to the full 1000-step (20 s) episode length on all 20 evaluation rollouts, evaluated on identical seeds via evaluate.py. The PPO GIF rollout includes a mid-run angular-velocity poke at t = 8 s that the policy absorbs without losing balance. The swing-up experiment (swingup.py) catches the pole at t = 6.4 s under both LQR and PPO catch controllers and holds it through the end of the episode.

USE CASES & APPLICATIONS

The inverted-pendulum-on-cart is the canonical low-order stand-in for a broad family of underactuated balance problems: rocket booster landing (thrust-vectored balance about an unstable equilibrium), Segway/self-balancing robots, crane anti-sway control (the load swings like the pole while the trolley plays the cart), and the single-support phase of humanoid balance. What this project demonstrates specifically is that a policy trained against a physically faithful simulator — deadband, quantization, friction, and impact all included — matches a classically-derived optimal controller (LQR) on its own turf, without hand-tuning, while also generalizing across the swing-up/catch mode switch that LQR alone can’t handle (LQR is only valid near the linearization point; the energy-pump hybrid is required to bridge from hanging). That’s the argument for RL on this class of plant: not that it beats LQR near equilibrium, but that it absorbs modeling nastiness and mode transitions without a fresh derivation each time.

FILES & REPRODUCTION

• cartpole_env.py — custom Gymnasium environment: all physics, quantization, deadband, wall collisions
• lqr_baseline.py — linearization, continuous-time Riccati solve, LQR gains and evaluation
• swingup_ctrl.py — importable, side-effect-free energy-pump swing-up controller
• swingup.py — swing-up + catch experiment and animation
• train_ppo.py — PPO training via Stable-Baselines3
• evaluate.py — PPO-vs-LQR head-to-head table and balance GIF
• live_demo.py — real-time 50 Hz interactive demo with a push panel and results sheet
• ppo_balance.zip — trained policy checkpoint
• docs/PLAN.md — full sim2real roadmap (physical rig, system ID, domain randomization)

pip install -r requirements.txt
python lqr_baseline.py      # sanity check: LQR balances 100%
python train_ppo.py         # train PPO yourself (~6 min CPU), or use ppo_balance.zip
python evaluate.py          # PPO-vs-LQR table + balance GIF
python swingup.py           # swing-up + LQR catch (CATCH=ppo for PPO catch)
python live_demo.py         # the interactive demo