DownloadERNIE-4.5-21B-A3B-Base-PT is a post-trained text-only Mixture-of-Experts (MoE) model from Baidu’s ERNIE 4.5 series, featuring 21 billion total parameters and 3 billion activated per token. It is designed to excel in general-purpose language understanding and generation, refined through post-training techniques like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Unified Preference Optimization (UPO). The model benefits from a staged pretraining process focused on building deep language capabilities before integrating multimodal elements. Although it originates from a joint multimodal training pipeline, this variant isolates only the text components for focused performance and easier deployment. It is compatible with the Transformers library and supports long-context processing up to 131,072 tokens. The model also integrates smoothly with PaddlePaddle, FastDeploy, and vLLM inference, enabling scalable deployment across various platforms.
Features
- 21B parameters with 3B active per token using MoE architecture
- Post-trained for text generation using SFT, DPO, and UPO
- Supports long contexts up to 131,072 tokens
- Fine-tuning ready with ERNIEKit (LoRA, multi-GPU, DPO)
- Compatible with Hugging Face Transformers and vLLM
- High inference efficiency via quantization and load balancing
- Staged training enhances stability and language depth
- Deployable with FastDeploy for scalable service integration
Project Samples
