DownloadERNIE-4.5-21B-A3B-Base-Paddle is a powerful text-focused Mixture-of-Experts (MoE) model developed by Baidu with 21 billion total parameters and 3 billion activated per token. It is pretrained using a staged approach that first builds strong language understanding and long-text capabilities before integrating broader modality alignment. While vision components were used in joint training, this specific variant extracts only the text-related parameters, making it a lightweight but capable base model for natural language tasks. Its MoE architecture includes 64 text and 64 vision experts, with six of each activated, supported by shared experts for better generalization. Built on PaddlePaddle, the model benefits from memory-efficient pipeline scheduling, FP8 mixed-precision training, and advanced quantization strategies. It supports long contexts of up to 131,072 tokens, making it suitable for tasks requiring deep document understanding.
Features
- 21B parameter model with 3B active per token for high efficiency
- Text-only MoE base extracted from multimodal pretraining
- 64 text experts and 6 activated for each generation step
- Supports instruction tuning with ERNIEKit (SFT, LoRA, DPO)
- Pretrained using staged, scalable infrastructure
- Long context support up to 131,072 tokens
- Optimized for Chinese and English text generation
Project Samples
