Data-Driven Control of Inverted Pendulum Systems Using Reinforcement Learning

Author: Thirugnanam V S (Roll No: 24BVD1004)
Programme: B.Tech VLSI Design & Technology, VIT Chennai (2024–2028)
Supervisor: Dr. Soumya Ranjan Mahapatro, Associate Professor, SENSE, VIT Chennai
Submitted to: 2026 IEEE 12th Power India International Conference (PIICON) — Paper ID: 1170

---

Overview

This repository contains the complete implementation of a data-driven reinforcement learning framework for stabilising an inverted pendulum (cart-pole) system without an explicit mathematical model.

Two deep RL agents — DDPG and PPO — are trained inside a custom MATLAB environment with full nonlinear Lagrangian dynamics and RK4 integration. A novel Safety-Aware Reward Shaping (SARS) function is introduced with six penalty terms, including a cart-velocity penalty that eliminates the degenerate drift solution observed in prior reward formulations.

Key Results

Metric	DDPG	PPO + SARS	LQR Baseline
Training episodes	273	229	N/A
Training time	3m 49s	1m 32s	<1s
Safety violations	0	0	0
Peak pole angle	2.87°	3.80°	2.87°
Max cart displacement	0.78m	0.27m	0.20m
RMS control effort	0.28N	0.21N	0.34N

Both RL agents outperform LQR on energy efficiency while requiring no system model.

---

Novel Contributions

1. SARS Reward Function — Six-term weighted reward:

   r(s,a) = w₁ − w₂θ² − w₃x² − w₄F² − w₅V(s) − w₆ẋ²
   

   The velocity penalty w₆ẋ² is novel — it eliminates the degenerate drift-and-balance solution by penalising cart motion at every timestep, not just at the destination.

2. Observation Normalisation — All states normalised to [−1, +1] before network input, resolving PPO gradient imbalance caused by the 3× scale difference between cart position (±4.8m) and pole angle (±π/2 rad).

3. Five-Phase Smooth Curriculum — Progressive difficulty increase with a maximum 2× jump per phase, preventing DDPG critic divergence observed with the original 3-phase design.

4. Degenerate Drift Diagnosis — Systematic identification and fix of the drift-and-balance failure mode in cart-pole RL.

---

Repository Structure

├── matlab/
│   ├── step0_create_params.m       # Run this FIRST — creates cart_pole_params.mat
│   ├── CartPoleEnv.m               # Custom RL environment (v6b, 5-phase curriculum)
│   ├── pid_baseline_04.m           # LQR baseline controller
│   ├── train_ddpg_v6.m             # Train DDPG agent (~4 min)
│   ├── train_ppo_v6.m              # Train PPO agent  (~2 min)
│   ├── compare_results_v6.m        # Evaluate all 3 controllers + save metrics
│   ├── generate_white_figures.m    # Generate publication-quality PNG figures
│   └── animate_pendulum_v6.m       # Animate trained agents (no toolbox needed)
│
├── latex/
│   ├── main.tex                    # Complete IEEE paper LaTeX source
│   ├── references.bib              # Bibliography (10 references)
│   ├── fig1_ddpg.png               # DDPG training curves
│   ├── fig2_ppo.png                # PPO training curves
│   ├── fig3_lqr.png                # LQR baseline response
│   ├── fig4_trajectories.png       # Three-way state comparison
│   ├── fig5_convergence.png        # Overlaid training convergence
│   └── fig6_barchart.png           # Performance metrics bar chart
│
└── docs/
    └── FUTURE_WORK.md              # Planned extensions (IRL-PID, hardware)

---

How to Run

Requirements

• MATLAB R2024a or later (R2026a recommended)
• Reinforcement Learning Toolbox
• Deep Learning Toolbox
• Control System Toolbox
• No Image Processing Toolbox needed

Step-by-Step

% Step 1: Create parameters
run('matlab/step0_create_params.m')

% Step 2: LQR baseline (~5 seconds)
run('matlab/pid_baseline_04.m')

% Step 3: Train DDPG (~4 minutes)
run('matlab/train_ddpg_v6.m')

% Step 4: Train PPO (~2 minutes)
run('matlab/train_ppo_v6.m')

% Step 5: Evaluate all controllers
run('matlab/compare_results_v6.m')

% Step 6: Generate white publication figures
run('matlab/generate_white_figures.m')

% Step 7: Animate (optional)
run('matlab/animate_pendulum_v6.m')

Compile the Paper

cd latex/
pdflatex main.tex
bibtex main
pdflatex main.tex
pdflatex main.tex

---

System Model

The cart-pole nonlinear Lagrangian equations of motion:

$$\ddot{\theta} = \frac{(M+m)g\sin\theta - \cos\theta[F + m\ell\dot{\theta}^2\sin\theta - b\dot{x}]}{\ell(M+m-m\cos^2\theta)}$$

$$\ddot{x} = \frac{F + m\ell(\dot{\theta}^2\sin\theta - \ddot{\theta}\cos\theta) - b\dot{x}}{M+m}$$

Physical parameters: M=1.0kg, m=0.1kg, ℓ=0.5m, b=0.1N·s/m, Δt=0.02s

---

Curriculum Design

Five-phase smooth curriculum (max 2× difficulty increase per transition):

Phase	Episodes	Cart Reset Range
1	1–100	±0.05 m
2	101–200	±0.20 m
3	201–350	±0.40 m
4	351–500	±0.60 m
5	501–800	±0.80 m

The original 3-phase design caused DDPG critic Q-value divergence in every training run due to a 2.67× difficulty jump at episode 301. The 5-phase design resolved this.

---

Citation

If you use this work, please cite:

@inproceedings{thirugnanam2026sars,
  author    = {Thirugnanam, V S and Mahapatro, Soumya Ranjan},
  title     = {Data-Driven Control of Inverted Pendulum Systems
               Using Reinforcement Learning},
  booktitle = {2026 IEEE 12th Power India International Conference (PIICON)},
  year      = {2026},
  note      = {Paper ID: 1170}
}

---

Future Work

• Hardware validation on Quanser cart-pole (available at VIT Chennai SENSE lab)
• Extension to double inverted pendulum
• IRL-based PID synthesis from RL expert policies (following Jesawada et al., IEEE TASE 2026)
• Lyapunov-based formal safety certificates during training

---

Acknowledgment

This work was supervised by Dr. Soumya Ranjan Mahapatro, Associate Professor, School of Electronics Engineering (SENSE), VIT Chennai. MATLAB R2026a access provided by VIT Chennai.