ppo adaptive kl

PPO with Adaptive KL is a variant of Proximal Policy Optimization that dynamically adjusts the KL divergence penalty coefficient during training based on observed policy changes.

## What Is Adaptive KL in PPO?

• Mechanism: Increases penalty when KL exceeds target, decreases when below
• Target KL: Typically 0.01-0.02 for stable training
• Adaptation Rate: Usually 1.5× increase or 0.5× decrease per update
• Alternative: PPO-Clip uses hard clipping instead of adaptive penalty

## Why Adaptive KL Matters

Fixed KL coefficients either over-constrain learning (too high) or allow destructive updates (too low). Adaptive tuning maintains stable training across different phases.

# Adaptive KL coefficient update
target_kl = 0.01
kl_coef = 0.2  # Initial coefficient

for epoch in training:
    kl_div = compute_kl(old_policy, new_policy)
    
    if kl_div > 1.5 * target_kl:
        kl_coef *= 2.0  # Policy changing too fast
    elif kl_div < target_kl / 1.5:
        kl_coef *= 0.5  # Can be more aggressive
    
    # Clip to reasonable bounds
    kl_coef = np.clip(kl_coef, 0.0001, 10.0)

PPO-Clip (using clipped surrogate objective) has largely replaced adaptive KL in practice due to simpler implementation.