Download README.md (Crawlab)

Other Useful Business Software

Gemini 3 and 200+ AI Models on One Platform

Access Google's best plus Claude, Llama, and Gemma. Fine-tune and deploy from one console.

Build generative AI apps with Vertex AI Studio. Switch between models without switching platforms.

Start Free

Build on Google Cloud with $300 in Free Credit

New to Google Cloud? Get $300 in free credit to explore Compute Engine, BigQuery, Cloud Run, Vertex AI, and 150+ other products.

Start your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query exabytes in BigQuery, or build AI apps with Vertex AI and Gemini. Once your credits are used, keep building with 20+ products with free monthly usage, including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. Sign up to start building right away.

Start Free Trial

Fully Managed MySQL, PostgreSQL, and SQL Server

Automatic backups, patching, replication, and failover. Focus on your app, not your database.

Cloud SQL handles your database ops end to end. Migrate from on-prem or other clouds with free migration tools.

Try Free

Cut Your Data Warehouse Bill by 54%

Migrate from Snowflake, Redshift, or Databricks with free tools. No SQL rewrites.

BigQuery delivers 54% lower TCO with serverless scale and flexible pricing. Free migration tools handle the SQL translation automatically.

Try Free

Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.

Try free now

Crawlab Overview

Distributed web crawler admin platform for spiders management

Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. Please use docker-compose to one-click to start up. By doing so, you don't even have to configure MongoDB database. The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. Master node and worker nodes communicate with each other via gRPC (a RPC framework). Tasks are scheduled by the task scheduler module in the master node, and received by the task handler module in worker nodes, which executes these tasks in task runners. Task runners are actually processes running spider or crawler programs, and can also send data through gRPC (integrated in SDK) to other data sources, e.g. MongoDB.

Additional Details for Crawlab

Operating Systems

Linux, Mac, Windows

Programming Language

Go, Java, PHP, Python

Registered

2023-01-05

Last Updated

2023-07-26

Maintainers

sf-editor1

Keep Me Updated!

Get an email when this software releases a new version (no other emails will be sent). Get newsletters and notices that include site news, special offers and exclusive discounts about IT products & services.

I understand by clicking on "Subscribe" below I am agreeing to the SourceForge Terms of Use and the Privacy Policy which describe how we use and share your data. I agree to receive these communications from SourceForge.net. I understand that I can withdraw my consent at anytime.

JavaScript is required for this form.

Crawlab