Magnetic Levitation, LQR-Primed RL vs Magnet Drift

A steel ball suspended under an electromagnet, the classic fast unstable SISO plant. A discrete LQR is compared against a behaviour-cloned-then-PPO-fine-tuned policy across magnet-constant drift, taps, and a live setpoint change.

OVERVIEW & MOTIVATION

Magnetic levitation is unstable through an inverse-square force law rather than gravity torque, giving an open-loop pole around +44 rad/s at the nominal 10 mm gap and forcing a 1 kHz control loop. The question this project asks is not just “can either controller balance the ball” but which one survives the plant changing underneath it: coil heating drifts the magnet constant, setpoints get moved, and the gap gets tapped. PPO trained from scratch could not even learn to balance here, so the method itself is part of the result. Part of a five-plant series on RL for underactuated control.

PHYSICAL SYSTEM & PARAMETERS

State is [z, z_dot, i], the air gap between magnet and ball (growing downward), gap velocity, and coil current; the control input is coil voltage, with the electrical L di/dt lag modelled explicitly rather than assumed instantaneous. Force follows the inverse-square law m z_dd = m g - k_m (i/z)^2, integrated with RK4 at 1 kHz (4 substeps). The episode ends if the ball hits the magnet (z < 1 mm) or falls out of sensor range (z > 19 mm).

PARAMETER	VALUE	UNIT
BALL MASS (m)	0.02	kg
GRAVITY (g)	9.81	m/s2
MAGNET CONSTANT (k_m)	3.0e-5	N m2/A2
COIL RESISTANCE (R_coil)	2.0	ohm
COIL INDUCTANCE (L_coil)	0.05	H
DRIVE VOLTAGE (V_MAX)	12.0	V
NOMINAL OPERATING GAP (Z0)	10	mm
EQUILIBRIUM CURRENT (I0)	~0.81	A
CONTROL PERIOD (DT)	1 (RK4, 4 substeps)	kHz
GAP SENSOR RANGE	0-20	mm
GAP ADC RESOLUTION	10-bit (1024 counts)	-
GAP SENSOR NOISE	50	um rms
CURRENT SENSE NOISE	5	mA rms
EPISODE LENGTH	4000 steps / 4	s
TERMINATION GAP BOUNDS	1 to 19	mm

METHOD

Observations are normalized to O(1) rather than raw metres: [(z-Z0)/Z0, zd_filt, (i-I0)/I0]. Raw-metre inputs forced the network toward gains of roughly 4000 per input unit, and behaviour cloning could not reach the needed precision until inputs were scaled. Gap velocity is not directly measurable; the observation instead reports a first-order low-pass filtered finite-difference of the noisy, quantized gap reading, matching what real firmware would compute. The action is a scalar in [-1, 1] mapped to [0, V_MAX] through a unipolar drive (V = (a+1)/2 * V_MAX). Domain randomization scales k_m by +/-20% per episode to emulate coil heating drift.

The LQR baseline linearizes the nonlinear inverse-square force model numerically about the nominal operating point and solves the discrete algebraic Riccati equation at 1 kHz.

PPO from scratch failed twice: episode length plateaued near 90 ms across 3,000,000 timesteps with tuned gamma and exploration noise, because the open-loop plant diverges in tens of milliseconds and random exploration dies before credit assignment begins. The fix is a stabilizing prior: pretrain_bc.py clones the LQR into the policy network via 25 epochs of supervised regression on 300,000 noisy demonstrations, then train_ppo.py —finetune continues with PPO under k_m randomization (n_steps=1024, batch_size=2048, learning_rate=3e-4 falling to 1e-4, gamma=0.998, gae_lambda=0.97, ent_coef=0.0).

RESULTS

Balance = hold the ball 4 s; RMS gap error computed over the last 3.5 s; 10 episodes per cell, both controllers keeping their nominal design while k_m is scaled:

k_m SCALE	LQR HELD	LQR RMS (mm)	BC+PPO HELD	BC+PPO RMS (mm)
0.70	10/10	0.77	10/10	1.24
0.85	10/10	0.31	10/10	0.62
1.00	10/10	0.04	10/10	0.12
1.15	7/10	0.27	10/10	0.37
1.30	5/10	0.46	10/10	0.87

Tap test (z_dot += 5 cm/s): LQR 10/10, BC+PPO 10/10. Live setpoint change 10 mm to 8 mm: the LQR crashed (its linearization is 20% wrong at the new gap even with operating-point scheduling); BC+PPO settled in 0.07 s, its k_m randomization having taught it the family of plants a setpoint shift creates. At the nominal point the LQR is roughly 3x more accurate (0.04 mm vs 0.12 mm RMS); away from nominal it is the one that starts dropping the ball.

USE CASES & APPLICATIONS

The gap-holding problem here is the core control loop behind magnetic bearings, maglev transport levitation, and any application that needs contactless, precision positioning against a nonlinear, temperature-drifting actuator. Robustness to parameter drift under a fixed setpoint and under setpoint changes is exactly the requirement for precision lithography stages and contactless wafer or component handling, where a controller that degrades gracefully as coil or magnet properties shift is worth more than one that is only optimal at a single nominal point.

FILES & REPRODUCTION

maglev_env.py     custom Gymnasium env (electrical lag, sensor model, k_m rand)
lqr_baseline.py   numeric linearization -> discrete ARE at 1 kHz
pretrain_bc.py    LQR demonstrations -> behaviour cloning warm start
train_ppo.py      PPO (from scratch = documented failure; --finetune = the fix)
evaluate.py       k_m robustness sweep, tap test, GIF
live_demo.py      real-time interactive demo (--headless N to verify)

pip install -r requirements.txt
python lqr_baseline.py          # discrete LQR at 1 kHz: 20/20, 0.04 mm RMS
python pretrain_bc.py           # clone LQR into the policy (~2 min)
python train_ppo.py --finetune  # PPO fine-tune under k_m randomization
python evaluate.py              # robustness sweep table + GIF
python live_demo.py             # interactive demo