Download
CogVLM is an open-source visual–language model suite—and its GUI-oriented sibling CogAgent—aimed at image understanding, grounding, and multi-turn dialogue, with optional agent actions on real UI screenshots. The flagship CogVLM-17B combines ~10B visual parameters with ~7B language parameters and supports 490×490 inputs; CogAgent-18B extends this to 1120×1120 and adds plan/next-action outputs plus grounded operation coordinates for GUI tasks. The repo provides multiple ways to run models (CLI, web demo, and OpenAI-Vision–style APIs), along with quantization options that reduce VRAM needs (e.g., 4-bit). It includes checkpoints for chat, base, and grounding variants, plus recipes for model-parallel inference and LoRA fine-tuning. The documentation covers task prompts for general dialogue, visual grounding (box→caption, caption→box, caption+boxes), and GUI agent workflows that produce structured actions with bounding boxes.
Features
- Pretrained VLMs (CogVLM-17B) and GUI-capable CogAgent-18B
- Multi-turn image dialogue, visual grounding, and GUI action planning
- Ready-to-use demos: CLI (SAT/HF), Gradio web UI, and OpenAI-Vision–style API
- Checkpoints for chat/base/grounding variants with prompt templates
- Quantization support (4-bit/8-bit) and model-parallel inference on multi-GPU
- Fine-tuning examples (e.g., CAPTCHA) and evaluation scripts
Project Samples
