Moonlight
Moonlight is an open-weight 16B MoE language-model family released with Moonshot research on scaling the Muon optimizer. It activates roughly 3B parameters per token and includes pretrained, instruction-tuned, and intermediate checkpoints. The research identifies optimizer adjustments for stable large-scale language-model training and provides a Muon implementation. Moonlight also serves as the language backbone for Kimi-VL. Its inclusion reflects a substantive, independently trained research model, with MIT weights available for replication and downstream studies.
2025-02-23
16B total, approximately 3B activated
Mixture of Experts (MoE) Transformer
MIT
Specifications
- Parameters
- 16B total, approximately 3B activated
- Architecture
- Mixture of Experts (MoE) Transformer
- License
- MIT
- Type
- text
- Modalities
- text
Benchmark Scores
Advanced Specifications
- Model Family
- Kimi
- API Access
- Not Available
- Chat Interface
- Not Available
- Variants
- Moonlight-16B-A3BMoonlight-16B-A3B-InstructIntermediate training checkpoints
Capabilities & Limitations
- Capabilities
- language modelinginstruction followingoptimizer research
- Known Limitations
- A research-scale model rather than Moonshot’s hosted flagship.Reported optimizer comparisons depend on the paper’s matched training setup.
- Notable Use Cases
- Muon training researchopen-model replicationdownstream fine-tuning