🕸️

GPT-Crawler

Technical SEO

👁️ 2 views

Crawler that crawls websites to generate knowledge files from web content for custom GPTs and LLM training

Visit Tool →

💰 Pricing

Open-source

Billing: one-time

💻 Platform

cli, api

👤 Target User

intermediate

Company: small-team,agency

📜 License

MIT

🔌 Integrations

openai,builder-io

🔄 Last Update

12/15/2025

API Available Data Export 🔥 Trending

⭐ 18,500

Stars

👥 22,000

Users

⭐ 4.7

Rating

🏷️ Tags

crawler ai-training llm data-extraction

Similar Tools & Alternatives

View All (236) →

📦

Puppeteer

Node library to control headless Chrome for complex SPA SEO rendering tests

🛠️

Hoppscotch

Open-source API development ecosystem used by SEOs to test and debug headless CMS endpoints and schema APIs

🕸️

Scrapy

Powerful open-source web crawling framework for Python developers to build custom SEO scrapers

📦

Meilisearch

Lightning fast open-source search engine API to improve site-search SEO and UX

🚀

Astro

The web framework for content-driven websites with built-in SEO optimizations and API routes

🏗️

Workbox

JavaScript libraries for adding offline support and power to Progressive Web Apps

← Back to list