Two-Wheel Balancing Robot, Velocity Tracking

A planar two-wheel inverted-pendulum robot (Segway architecture) that balances while following randomly stepping velocity commands, modeled as a 1 kg hobby build with torque deadband, coarse odometry, and IMU resolution. Over 20 episodes, a 2M-step PPO tracks velocity to 0.123 m/s mean absolute error against the discrete LQR’s 0.163 m/s — about 25% tighter, since the LQR carries steady-state error under the deadband with no integral action. The trade is honest: the LQR survives 20/20 and recovers every shove, PPO 19/20, so neither dominates at this training budget.

OVERVIEW & MOTIVATION

TWIP is the fifth plant in a five-plant series on RL for underactuated control (cart-pole, Furuta pendulum, reaction-wheel pendulum, TWIP, ballbot), sharing a common sim2real recipe across all five. The interesting coupling here: the same wheel torque that drives the robot forward is the torque that tips it over, with opposite sign, so the robot has to lean into a motion before it can execute it — the classic Segway control problem. The task adds velocity tracking on top of balance: the robot must follow a randomly stepping commanded velocity, not just stay upright, so the observation includes the tracking error v - v_cmd.

PHYSICAL SYSTEM & PARAMETERS

Modeled as a ~1 kg hobby robot: geared TT motors drive both wheels together (planar model, single combined torque channel), body pivots on the wheel axle. State is [x, v, theta, theta_dot], theta being body lean from vertical. Dynamics are the standard planar TWIP mass-matrix formulation (no wheel slip), where torque enters the body equation with opposite sign from the drivetrain equation — that’s the segway coupling. Realism modeled in twip_env.py: torque limit and deadband, viscous friction on both the ground contact and body pivot, wheel-odometry quantization (20 CPR x4 encoder discs) and IMU angle quantization, optional actuation delay and domain-randomization hooks, RK4 integration at 100 Hz. Energy conservation verified to 3e-6 over 10 s with dissipation disabled.

PARAMETER	VALUE	UNIT
BODY MASS (m_b)	0.80	KG
AXLE-TO-BODY-COM LENGTH (l)	0.10	M
WHEEL MASS, BOTH WHEELS (m_w)	0.10	KG
WHEEL RADIUS (r)	0.033	M
GRAVITY (g)	9.81	M/S^2
TRANSLATIONAL FRICTION (b_x)	0.35	N·S/M
BODY PIVOT FRICTION (b_th)	2e-4	N·M·S/RAD
MAX WHEEL TORQUE (TAU_MAX)	0.40	N·M
TORQUE DEADBAND	0.02	N·M
CONTROL PERIOD (DT)	0.01 (100)	S (HZ)
RK4 SUBSTEPS PER STEP	5	—
VELOCITY COMMAND RANGE (V_MAX)	1.2	M/S
WHEEL ODOMETRY RESOLUTION	20 x4 = 80	COUNTS/REV
IMU ANGLE RESOLUTION	0.1	DEG
VELOCITY COMMAND STEP INTERVAL	3–6	S
EPISODE LENGTH	2000 (20)	STEPS (S)
FALL THRESHOLD	0.6 (~35)	RAD (DEG)

METHOD

Observation space: Box([-50,-5,-pi,-30,-2.4], [50,5,pi,30,2.4]), i.e. [x, v, theta, theta_dot, v - v_cmd] (x and v are quantized odometry/angle readings; the tracking error term is appended raw). Action space: Box(-1, 1) scaled to +/- TAU_MAX, total wheel torque split across both motors. Reward: cos(theta) - 0.5*(v - v_cmd)^2 - 0.02*a^2 - 0.01*theta_dot^2, with a -5.0 penalty on falling. Episode terminates when |theta| > 0.6 rad (~35 deg), truncates at 2000 steps (20 s). Velocity commands step to a new random value in [-V_MAX, V_MAX] every 300–600 steps (3–6 s).

LQR: 4-state discrete design over [x, v, theta, theta_dot], numerically linearized about upright and discretized via cont2discrete + solve_discrete_are. Q = diag([0.0, 8.0, 60.0, 1.0]) (x weight zero — position is free, only velocity tracking matters), R = [[10.0]]. Velocity tracking is implemented by feeding back v - v_cmd in place of raw v; no integral action is added, so steady-state error under the torque deadband is reported honestly rather than corrected.

PPO (train_ppo.py, Stable-Baselines3 MlpPolicy): 2,000,000 timesteps, 8 parallel envs, n_steps=512, batch_size=1024, learning_rate=3e-4, gamma=0.99, gae_lambda=0.95, ent_coef=0.005, seed 0. Training run takes roughly 15 minutes on CPU.

RESULTS

Evaluated over 20 episodes of 20 s each with random velocity commands:

METRIC	LQR (DISCRETE)	PPO (2M STEPS)
SURVIVAL	20/20	19/20
MEAN ABS. VELOCITY ERROR	0.163 M/S	0.123 M/S
SHOVE RECOVERY (theta_dot += 1.5 RAD/S)	20/20	19/20

PPO tracks commanded velocity about 25% tighter than LQR — the LQR carries steady-state error under the torque deadband because it has no integral action, while PPO’s learned policy compensates for it directly. The LQR is marginally more robust on survival and shove recovery. Neither controller dominates outright at this training budget, which the source material treats as the interesting result rather than a shortcoming to explain away.

Live interactive demo (live_demo.py) verified headless: a +0.8 m/s velocity step settled (within tracking and lean tolerance) in 0.68 s under PPO control.

USE CASES & APPLICATIONS

The Segway/hoverboard-class self-balancing vehicle is the direct real-world analogue of this plant — same lean-to-drive coupling, same need to track a commanded velocity while staying upright. The same balance-plus-tracking control problem also underlies warehouse and logistics balance robots (two-wheel AGV-style platforms navigating tight indoor spaces) and the base/torso balancing loop in wheeled humanoid robots, where a two-wheel inverted-pendulum base carries a manipulator or upper body. These are framed as applications the plant is representative of, not outcomes demonstrated on physical hardware — no Segway-class vehicle, warehouse robot, or wheeled humanoid was built or tested.

FILES & REPRODUCTION

twip_env.py       custom Gymnasium env (all physics here)
lqr_baseline.py   numeric linearization -> discrete ARE -> gains + eval
train_ppo.py      PPO training (Stable-Baselines3)
evaluate.py       LQR-vs-PPO table, shove test, GIF
live_demo.py      real-time interactive demo with drive slider + results sheet

Same electronics as the cart-pole rig: an Arduino Uno reads encoders and an MPU6050 IMU and streams at 100 Hz over serial while the PC runs the policy. The planar simulation maps to the real robot’s pitch axis; yaw would be a separate, simpler differential-torque loop.

pip install -r requirements.txt
python lqr_baseline.py    # discrete LQR: 20/20, prints gains
python train_ppo.py       # ~15 min CPU, saves ppo_twip.zip
python evaluate.py        # head-to-head table + results/twip_ppo.gif
python live_demo.py       # interactive demo (--headless N to verify)