Policy-gradient methods learn a robot's behavior directly — nudging the policy toward actions that earned more reward — and are the family behind the continuous-control algorithms that teach real robots to walk, fly, and manipulate.
Policy-gradient methods learn the behavior itself: try things, and make the actions that led to more reward more likely next time. Because the behavior can be smooth, they handle continuous controls like joint torques that value-table methods struggle with.
🎯 Quick challenge
A key advantage of policy-gradient methods over Q-learning is…
Q-learning learns how good each action is and then picks the best. Policy-gradient methods skip the middleman and learn the behavior itself — which turns out to matter enormously for real robots.
The idea
Represent the policy as a function (usually a neural network) with parameters θ that maps states to a distribution over actions. Then adjust θ to increase the probability of actions that led to high reward and decrease it for the rest. Formally, you ascend the gradient of expected return:
∇θ J ≈ Σ ∇θ log π(a|s) · A
└ make good actions more likely, weighted by how good ┘
where A (the advantage) measures how much better an action was than expected.
Reinforce what worked
Actions that beat expectations get more likely; actions that underperform get less likely. Repeat and the behavior improves.
Why robots need it
The decisive advantage: policy gradients handle continuous action spaces naturally. A robot's command is a vector of joint torques or velocities — there's no sensible "max over all actions" as DQN requires. A policy network just outputs the continuous action (often as a Gaussian). They also support stochastic policies, useful for exploration and for tasks with inherent randomness.
The trade-off
Vanilla policy gradients (REINFORCE) are high-variance and sample-hungry — rewards are noisy, so the gradient estimate is jumpy and learning is slow. Two fixes define modern RL: pairing the policy with a value estimate to reduce variance (actor-critic), and constraining how far the policy changes each step so training stays stable (PPO).
Why it matters
Policy-gradient methods are the backbone of continuous robot control with RL — the lineage (A2C, TRPO, PPO, SAC) that trains locomotion, dexterous manipulation, and drone control in simulation and transfers it to hardware.
