DDPG: how the pieces fit together

actAt = μ(St | θμ) + 𝒩t
targetyi = ri + γ Q′(si+1, μ′(si+1 | θμ′) | θQ′)
critic lossL(θQ) = 1NΣi(yi − Q(si, ai | θQ))2
actor gradient∇θμJ ≈ 1NΣi∇aQ(s, a | θQ)|s=si, a=μ(si) ∇θμμ(s | θμ)|si
soft updateθQ′ ← τ θQ + (1 − τ) θQ′,   θμ′ ← τ θμ + (1 − τ) θμ′
data flow action policy gradient soft update Environment Physical environment Sensors Actor μ(s|θμ) Critic Q(s,a|θQ) Target actor μ′ Target critic Q′ OU noise 𝒩 Replay buffer ℛ Random sampling St At At + 𝒩t (St, At, Rt, St+1) minibatch ∇aQ μ′(si+1) yi τ τ
Ready
Stage – of 9

Keyboard: Space or → for next stage, ← for previous, S to play one full step, P to play or pause.

Settings

Typical DDPG values are τ = 0.001 to 0.01.