Scrapy-Cluster
Distributed crawling system for massive SEO datasets using Redis and Kafka
Specifications
- Pricing
- open-source
- Price range
- $0
- Billing
- one-time
- License
- Apache
- Platform
- api, cli
- Languages
- en
- Target user
- expert
- Company size
- enterprise
- API
- Yes
- Data export
- Yes
- Last update
- 2025-05-15
Tags
Related tools
- 🚀 Aiohttp-Crawler
Asynchronous HTTP client for ultra-fast SEO crawling and API requests
- 🧪 Trafilatura
Python package and command-line tool to gather text and metadata on the web
- 🧪 Gevent-SEO-Crawler
High-performance concurrent crawler using gevent for large scale site audits
- 🗺️ Ultimate Sitemap Parser
Robust Python sitemap parser to extract URLs and detect errors in XML sitemap files
- 🚄 FastAPI-SEO-API
Template for building high-performance SEO microservices and custom internal tools
- 🧠 Spacy
Industrial-strength NLP for advanced content analysis and semantic keyword clustering