DownloadQwen2.5-1.5B-Instruct is an instruction-tuned variant of the Qwen2.5 language model with 1.54 billion parameters, designed for text generation and conversational tasks. It was developed for use within the Gensyn RL Swarm system, which enables decentralized reinforcement learning fine-tuning over peer-to-peer networks. The model architecture includes rotary positional embeddings (RoPE), SwiGLU activation, RMSNorm, attention QKV bias, and tied word embeddings. It features 28 layers, a GQA attention mechanism with 12 query heads and 2 key-value heads, and a context window of up to 32,768 tokens for input and 8,192 tokens for output. While optimized for RL Swarm use, it can be integrated into standard workflows for inference and chat once fine-tuned. It supports BF16 tensors and is distributed as a Safetensors model. The base model is Qwen2.5-1.5B, with this version enhanced for instruction following and dialogue.
Features
- Instruction-tuned for chat and task-oriented dialogue
- 1.54B total parameters with 1.31B non-embedding parameters
- Uses rotary position encodings (RoPE) and SwiGLU activation
- Includes RMSNorm and attention QKV bias
- 28 transformer layers with grouped-query attention (GQA)
- 32K token context length for input, 8K token generation length
- Compatible with Gensyn RL Swarm for decentralized RL fine-tuning
- Ready for use with Featherless AI inference or local deployment
Project Samples
