High-performance concurrent web crawler with real-time dashboard, AI-powered content analysis, and intelligent filtering.
I wanted to build a proper web crawler — not a quick script that fetches a few pages, but a production-quality system with concurrent workers, smart caching, and a real-time dashboard for monitoring what's happening.
The Go backend is the engine: configurable worker pools process pages in parallel, convert HTML to Markdown, and store everything with JSON metadata organised by domain. There's a priority queue for URL scheduling, time-based caching with automatic invalidation, and the ability to stop and resume crawls without losing progress. It respects robots.txt and does polite rate limiting.
The React/TypeScript frontend gives you a real-time dashboard showing crawl progress, domain details, and a page viewer for inspecting what was captured. There's also AI-powered content analysis that can summarise and categorise what the crawler finds.
The architecture separates concerns cleanly: the backend has modules for API handlers, caching, HTML-to-Markdown conversion, the crawler engine, a priority queue, and file system storage. Configuration supports environment variables, JSON config files, and in-UI settings. The whole thing runs in Docker.