Where the compute goes. Feedforward until long context

Where the compute actually goes. Feedforward, until long context.

Per token, per layer FLOPs inside one transformer block. The feedforward network is a fixed cost. Attention has a fixed projection cost plus a sequence term that grows with context length. Slide the context length and watch where the compute lives.

feedforward (fixed) attention projections (fixed) attention sequence term (grows)

Feedforward

1.074 G

Attn projections

0.537 G

Attn sequence

0.017 G

Attention share

Context length 8,192 tokens

Ordinary use is feedforward bound. At everyday context lengths most per token compute sits in the feedforward network, not attention. That is why mixture of experts, which enlarges the feedforward part, is where the parameter action is. Long context is a separate and more expensive regime, where the sequence term grows without bound and eventually dominates everything else.

FLOPs per token per layer with d_model 8192 and d_ff 32768, counting a multiply add as two FLOPs. Attention projections equal 2 times (4 d_model squared). Sequence term equals 4 seq d_model. Feedforward equals 2 times (2 d_model d_ff). The sequence term meets the feedforward term near 32768 tokens, where attention share crosses roughly sixty percent.