Read deeper.

of the values, where each weight comes from comparing the query with the corresponding key.

3.2Scaling the scores

We call this operation scaled attention. The input consists of queries and keys of dimension d_k and values of dimension d_v. We compute the dot products of the query with all keys, divide each by √d_k, and apply a softmax to obtain the weights on the values.

For small values of d_k the two forms behave alike, but as d_k grows the dot products grow large in magnitude, pushing the softmax into regions where it has extremely small gradients. Dividing every score by √d_k keeps them in a range where learning stays stable.

3.3Sparse blocks

Instead of attending to every position, each token attends only within a few blocks chosen by locality and content. This keeps memory linear in sequence length, and lets a single model read documents of up to a million tokens.

Blocks are recomputed at every layer, so information can still travel across the whole document. With a single block, the model reduces to local attention.

3.4Choosing blocks

We score candidate blocks with a small router that sees a summary of each block and the current token. The router is trained jointly with the model and adds less than one percent to the cost of a forward pass. In practice, eight blocks of 512 tokens recover nearly all of the accuracy of dense attention on our benchmarks.

On question answering over long documents, sparse attention stays within a point of dense attention while using a tenth of the memory. The gap narrows further as documents grow, since dense attention must truncate them.