proximal policy optimization

PPO (Proximal Policy Optimization) is the most widely used policy gradient RL algorithm — simplifying TRPO's constrained optimization into a clipped surrogate objective that achieves similar stability with much simpler implementation and better empirical performance.

PPO Clipped Objective

• Ratio: r_t(theta) = (pi_theta(a_t|s_t))/(pi_old(a_t|s_t)) — probability ratio between new and old policy.
• Clipped: L^CLIP = min(r_t A_t, clip(r_t, 1-epsilon, 1+epsilon) A_t) — clip the ratio to [1-epsilon, 1+epsilon].
• epsilon Parameter: Typically 0.1-0.2 — controls how much the policy can change per update.
• Mini-Batch: Multiple optimization epochs per data collection — more sample efficient than vanilla policy gradient.

Why It Matters

• Simplicity: Much simpler than TRPO — no conjugate gradient, no KL constraint, just clipping.
• RLHF: PPO is the standard algorithm for RLHF (Reinforcement Learning from Human Feedback) in LLMs.
• Versatility: Works for discrete and continuous actions, single and multi-agent, games and robotics.

PPO is the workhorse of modern RL — simple, stable, and effective policy optimization through clipped surrogate objectives.