A concurrent web crawler in Go with HTML-to-Markdown conversion, persistent queue, and a real-time dashboard.
Web Spider crawls websites concurrently, converts pages to Markdown, and serves everything through a REST API. The Go backend uses a worker pool architecture with configurable parallelism and a persistent queue so you can stop and resume crawls.
Each crawled page gets converted to Markdown with a custom algorithm. There's robots.txt compliance, a time-based cache with configurable expiration, and a priority queue for crawl ordering.
The real-time dashboard (Tailwind + shadcn styling) shows crawl progress, page counts, and lets you browse and visualise crawled data. There are separate HTML files for browsing pages and visualising the site structure.
The architecture is documented in PROJECT-ARCHITECTURE.md. Lefthook handles git hooks. Docker deployment is supported. This is the kind of infrastructure tool I find satisfying to build — take a complex concurrent problem and make it reliable.