NewsBite analysis · AI, Data, Programming · New Product, Software Release
MarkTechPost
A 125B Multimodal MoE With 6B Active Parameters Previewing the Qwen4 Architecture
Alibaba's Qwen team has released Qwen3.8-Flash-Next, an innovative multimodal model that cleverly uses a 125B backbone, with only 6B parameters activated per token. The architecture incorporates four significant improvements: a hybrid attention system, gated residual connections, an expanded N-gram embedding table, and a new training optimizer. It aims to lower operational costs while maximizing performance, achieving training costs roughly one-ninth that of its predecessor, Qwen3.7-Plus. As a preview of Qwen4, it indicates a shift towards more efficient, high-performing AI models.
Act nowConfidence: HighShort term horizon
Published 26 Aug 2026, 16:41 UTC3 min read
Visit original article ↗Lead image — MarkTechPost
Summary
TL;DR
- New model builds on a 125B backbone with 6B active parameters.
- Training cost reduced to about one-ninth of the previous model.
- Hybrid attention mechanism enhances performance.
- Features a large N-gram embedding table for improved contextual understanding.
- Supports fine-tuning and has high throughput capabilities.
Interesting facts
- The model can expand its context from 262K to 1M tokens using YaRN.
- Training employs both the Muon optimizer and traditional AdamW for better efficiency.
- Only 6B parameters activate per token, which is a significant technical achievement.