LearnDeep learning
Recurrent network
Give a network a memory that loops back on itself, and see how quickly that memory fades.
Drag the bars at the top to change the inputs. Click a step at the bottom to follow its influence through time.
How it works
A loop that remembers, briefly.
A recurrent network reads a sequence one step at a time. Its hidden state h is a running summary: each step mixes the new input with the previous state, hₜ = tanh(wₓxₜ + wₕhₜ₋₁), so information from early inputs can travel forward.
How much an early input still matters later is a chain of factors, wₕ × (1 − h²), one for every step in between. When those factors are below 1, the influence shrinks exponentially and is soon gone; above 1, it can grow until tanh saturates and squashes it again. The chart shows this influence step by step.
Training uses these same factors (backpropagation through time), so plain recurrent networks struggle to learn long-range patterns. LSTMs and GRUs add gates that keep memory alive for longer, and transformers avoid the loop altogether with attention.