Browsing: MoE

Three Chinese labs now hold the top of the open-weight leaderboard. Moonshot AI’s Kimi K3, DeepSeek V4 Pro, and Zhipu AI’s GLM-5.2 are all sparse Mixture-of-Experts…

Moonshot AI just released Kimi K3. It is a 2.8-trillion-parameter model with native vision and a 1-million-token context window. Moonshot calls it the world’s first open…

A German research consortium has published the pretraining report for Soofi S 30B-A3B. It is an open base model for German and English. Training ran end…

Thinking Machines Lab just released Inkling, their first model trained from scratch, weights are open, fine-tunable on Tinker. The lab pitches it as a base for…

Large hybrid MoE models like Nemotron-3-Super are accurate but expensive to serve. Their active parameters, KV cache, and Mamba state cap how many users a node…

Large hybrid MoE models like Nemotron-3-Super are accurate but expensive to serve. Their active parameters, KV cache, and Mamba state cap how many users a node…

Tencent’s Hy team released Hy3. Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model. It activates only 21B parameters per token. The weights ship under the Apache License…

Meituan has released LongCat-2.0, a large-scale Mixture-of-Experts (MoE) language model. It carries 1.6 trillion total parameters and activates about 48 billion per token. The model targets…

Prime Intellect has released prime-rl version 0.6.0. The framework targets reinforcement learning on trillion-parameter Mixture-of-Experts (MoE) models. It focuses on heavy agentic workloads, like long-horizon software-engineering…

MiniMax released MSA (MiniMax Sparse Attention), a sparse attention method built directly on Grouped Query Attention (GQA). It targets one bottleneck: the quadratic cost of softmax…