MiniMax released MSA (MiniMax Sparse Attention), a sparse attention method built directly on Grouped Query Attention (GQA). It targets one bottleneck: the quadratic cost of softmax attention at long context. The MiniMax research team tested it inside a 109B-parameter Mixture-of-Experts model trained with native multimodal data. They also open-sourced an inference kernel and shipped a production model, MiniMax-M3.
What is MSA (MiniMax Sparse Attention)
MSA (MiniMax Sparse Attention) factors attention into two stages: an Index Branch and a Main Branch. The Index Branch decides which key-value blocks each query should read. The Main Branch then runs exact softmax attention over only those blocks.
Selection happens at block granularity, not per token. The default block size is Bk = 128 tokens. Each query and GQA group keeps k = 16 blocks. That fixes the per-query budget at kBk = 2,048 key-value tokens.
The two cost structures differ. Dense GQA attention scales per query as O(N), the full context. MSA scales as O(kBk), which stays fixed as N grows. The compute gap therefore widens as context length increases.
Selection is shared inside each GQA group but independent across groups. One key-value head serves several query heads, and they share one block set. Different groups can attend to different long-range regions.
How the Two Branches Work
The Index Branch adds only two projection matrices to a standard GQA layer. It defines one index query head per GQA group and one shared index key head. It scores visible key tokens, then max-pools those scores to the block level.
A Top-k operator then selects the highest-scoring blocks per query and group. The local block containing the query is always included. This prevents the selector from dropping the query’s immediate neighborhood.
The Main Branch gathers causally visible tokens from the selected blocks. It applies scaled dot-product softmax attention restricted to those tokens. Each query head keeps its own query projection but shares the group’s block set.
A visualization in the report shows what the learned indexer selects. Heads concentrate on the local diagonal and the first block. They reserve the rest of the budget for a few long-range stripes.
https://arxiv.org/pdf/2606.13392v1
https://arxiv.org/pdf/2606.13392v1
How MSA is Trained
Top-k selection is non-differentiable, so the language-modeling loss cannot train the index projections. MSA solves this with a KL alignment loss. The loss matches the Index Branch distribution to the Main Branch attention pattern. The teacher is the group-averaged Main Branch distribution over the selected tokens.
Three mechanisms stabilize sparse training. Gradient Detach applies stop-gradient to the Index Branch input. This confines the KL loss to the index projections, not the backbone. Without it, larger KL coefficients caused gradient spikes and loss divergence.
Indexer Warmup runs full attention in both branches for the first iterations. The indexer learns from the KL loss before it controls routing. The forced Local Block reserves one slot for nearby context.
Ablations shaped the final recipe. An early variant added an Index Branch value head with its own output. Once warmup is used, that value head is no longer necessary. The final design drops it on efficiency grounds.
MSA supports two training routes. MSA-PT trains from scratch after a 40B-token indexer warmup. MSA-CPT converts a dense GQA checkpoint trained on 2.6T tokens. It then continues for 400B tokens, including 40B tokens of warmup.
The Kernel Co-Design
Theoretical sparsity does not become speed without a matching GPU path. MSA pairs the algorithm with two kernel ideas.
The first is exp-free Top-k selection. Softmax preserves order, so ranking raw scores yields identical indices. The kernel skips the max, exp, and sum steps before selection. At 128K context with k = 16, it ran 5.1× faster than torch.topk. It also beat the TileLang radix-select kernel by 3.7×.
The second is KV-outer sparse attention with query gather. Iterating over KV blocks raises arithmetic intensity versus iterating over queries. The kernel packs ⌈128/G⌉ query positions into one 128×128 score MMA. A two-phase forward splits the attention and combine steps across CTAs.
The open-source kernel, fmha_sm100, targets NVIDIA SM100 GPUs. It ships dense FlashAttention plus sparse Top-k kernels under an MIT license. It supports BF16, FP8, NVFP4, and FP4 precision.
How MSA Compares To Other Sparse Methods
The research team positions MSA against four natively trained sparse designs.
The table below summarizes the differences it describes.
MethodBackboneSelection granularityIndexer / selection signalMSAGQABlock-level (B_k = 128), per-GQA-group Top-kKL alignment lossNSAMQA / MHACompressed + selected blocks + sliding windowNative (end-to-end) trainingInfLLM-V2Dense↔sparse switchableParameter-free block selection + sliding windowParameter-free (no trained indexer)MoBAGQAVery large KV blocks (block-averaged keys)LM gradient onlyDSAMLA (MQA mode)Token-level; single Top-k shared across headsReLU lightning indexer
MSA’s distinguishing pair is per-GQA-group Top-k sharing combined with block-level selection. This keeps KV reads contiguous while giving each group its own retrieval.
The quality side holds up. Both sparse models stay broadly competitive with the Full-Attention baseline.
The table below shows representative results under the 3T-token budget.
BenchmarkFullMSA-PTMSA-CPTMMLU67.067.266.8GSM8K76.277.773.7HumanEval61.064.057.9RULER-8K79.884.277.2RULER-32K75.077.575.7VideoMME41.1145.4839.65
After long-context extension, MSA-CPT stayed close to Full on HELMET-128K and RULER-128K. Each query still attends to only 2,048 key-value tokens.
Explainer Playground
“;
for(var q=0;q0;c++)chosen[blk]!==undefined) continue;
chosen[blk]=’long’; need–;
html+=”;
for(var col=0;colq){ color=”#0d0d0d”; } // future (causal mask)
else if(chosen[col]===’sink’){ color=”#76B900″; }
else if(chosen[col]===’local’){ color=”#4a8f00″; }
else if(chosen[col]===’long’){ color=”#2e6fd6″; }
else { color=”#1d1d1d”; }
var bd = (col>q)?’#0d0d0d’:’#0e0e0e’;
html+=”;
}
html+=”;
}
html+=”;
root.querySelector(‘#mask’).innerHTML=html;
}
function fmtFlops(x){
if(x>=1e9) return (x/1e9).toFixed(1)+’ GFLOP’;
if(x>=1e6) return (x/1e6).toFixed(1)+’ MFLOP’;
if(x>=1e3) return (x/1e3).toFixed(1)+’ kFLOP’;
return Math.round(x)+’ FLOP’;
}
function commas(n){ return n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ‘,’); }
function updateCompute(idx){
var N=Ns[idx];
// per-token attention FLOPs (divide eq.12 totals by N)
var gqa = 2*Hq*dh*N; // dense GQA per token
var msa = didx*Hkv*N + 4*Hq*dh*k_budget*Bk; // index + main per token
var ratio = gqa/msa;
root.querySelector(‘#nval’).textContent=Nlabels[idx];
root.querySelector(‘#redux’).innerHTML=ratio.toFixed(1)+’×’;
root.querySelector(‘#gqaf’).textContent=fmtFlops(gqa);
root.querySelector(‘#msaf’).textContent=fmtFlops(msa);
var msaPct=Math.max(4, (msa/gqa)*100);
root.querySelector(‘#gqabar’).style.width=”100%”;
root.querySelector(‘#msabar’).style.width=msaPct.toFixed(2)+’%’;
root.querySelector(‘#budget’).textContent=commas(k_budget*Bk);
root.querySelector(‘#densetok’).textContent=commas(N);
postHeight();
}
// —- wiring —-
bslide.addEventListener(‘input’,function(){
B=+this.value; bval.textContent=B;
if(k>B) { k=B; kslide.value=B; kval.textContent=B; }
kslide.max=Math.min(8,B);
buildMask(); postHeight();
});
kslide.addEventListener(‘input’,function(){
k=+this.value; kval.textContent=k; buildMask(); postHeight();
});
tabs.forEach(function(t){
t.addEventListener(‘click’,function(){
tabs.forEach(function(x){x.classList.remove(‘on’)});
t.classList.add(‘on’); group=+t.getAttribute(‘data-g’); buildMask();
});
});
root.querySelector(‘#nslide’).addEventListener(‘input’,function(){ updateCompute(+this.value); });
// —- auto-resize for WordPress iframe —-
function postHeight(){
try{
var h=document.body.offsetHeight+40;
if(window.parent && window.parent!==window){
window.parent.postMessage({msaDemoHeight:h},’*’);
}
}catch(e){}
}
buildMask();
updateCompute(5);
window.addEventListener(‘load’,postHeight);
window.addEventListener(‘resize’,postHeight);
})();
