Documentation

README

WebSpider

A high-performance concurrent web crawler with a real-time dashboard and AI-powered content analysis. Crawl websites, convert pages to Markdown, and intelligently filter noise using Claude.

WebSpider Dashboard

Features

Concurrent Crawling -- Configurable worker pool (1-10+) with per-domain rate limiting and robots.txt compliance
HTML to Markdown -- Automatic conversion with metadata extraction, code block preservation, and table formatting
Smart Caching -- MD5-based URL caching with configurable TTL (default 24h) to avoid redundant requests
AI Content Filtering -- Claude-powered duplicate detection, title pattern analysis, and intelligent noise removal
Real-time Dashboard -- Live stats, Chart.js visualizations, domain browser, and activity log
Headless Chrome -- ChromeDP integration for JavaScript-rendered pages (SPAs, React apps)
Resume Capability -- Persistent queue saved to JSON survives crashes and restarts
Priority Queue -- Depth-based ordering crawls shallow pages first (breadth-first)

Architecture

graph TB subgraph Frontend["Frontend :3001"] UI[React 19 + TypeScript] Charts[Chart.js Visualizations] AIWizard[AI Analysis Wizard] UI --> Charts UI --> AIWizard end subgraph Backend["Go Backend :8082"] API[Gin REST API] Crawler[Crawler Engine] Queue[Priority Queue] Cache[URL Cache] Conv[HTML→Markdown] Browser[Headless Chrome] API --> Crawler Crawler --> Queue Crawler --> Cache Crawler --> Conv Crawler --> Browser end subgraph AI["AI Service"] Claude[Claude API] Patterns[Pattern Detection] Filters[Smart Filtering] end subgraph Storage["File Storage"] Pages[Markdown Pages] Meta[Metadata JSON] Raw[Raw HTML] QueueFile[Persistent Queue] end UI -->|REST API| API API --> Claude Browser -->|HTTP/Chrome| Web((Target Websites)) Crawler --> Pages Crawler --> Meta Crawler --> Raw Queue --> QueueFile

Component	Technology
Backend	Go 1.24, Gin, ChromeDP, goquery
Frontend	React 19, TypeScript, Vite 5, Tailwind CSS, shadcn/ui
AI	Claude API (Anthropic)
Storage	File system -- Markdown, JSON metadata, raw HTML

Quick Start

bash

# Prerequisites: Go 1.21+, Node.js 18+, npm

# Clone and install
git clone https://github.com/jonathanleahy/webspider.git
cd webspider
make install-deps

# Start development (hot reload for both services)
./dev-all.sh

# Or start individually
make backend-dev   # Backend on :8082
make frontend-dev  # Frontend on :3001

Access:

Frontend: http://localhost:3001
Backend API: http://localhost:8082/api/v1/

Configuration

Defaults in backend/pkg/config/config.go -- override via JSON config file:

bash

./bin/web-spider -config config.json

json

{
  "server": { "port": "8082", "host": "0.0.0.0" },
  "crawler": {
    "max_workers": 5,
    "max_depth": 50,
    "cache_duration": "24h",
    "rate_limit": "1s",
    "respect_robots": true,
    "user_agent": "WebSpider/1.0",
    "use_headless": true
  }
}

For AI features, set your API key:

bash

echo "ANTHROPIC_API_KEY=sk-ant-..." > backend/.env

API

All endpoints are under /api/v1/. Key routes:

Method	Endpoint	Description
`POST`	`/crawl/start`	Start crawling a URL
`POST`	`/crawl/stop`	Stop the crawler
`GET`	`/crawl/stats`	Live statistics
`PUT`	`/crawl/config`	Update workers, depth, rate limit at runtime
`GET`	`/domains`	List all crawled domains
`GET`	`/pages/:domain`	List pages for a domain
`GET`	`/page?domain=X&page=Y`	Get page content (Markdown + HTML + metadata)
`POST`	`/ai/analyze-duplicates`	AI duplicate detection
`POST`	`/ai/apply-filters`	Apply AI-suggested filters

See the User Guide for the full API reference.

Project Structure

shell

webspider/
├── backend/
│   ├── cmd/main.go              # Entry point
│   ├── internal/
│   │   ├── api/                 # REST API (Gin)
│   │   ├── ai/                  # Claude AI integration
│   │   ├── browser/             # Headless Chrome
│   │   ├── cache/               # URL caching
│   │   ├── converter/           # HTML-to-Markdown
│   │   ├── crawler/             # Core engine
│   │   ├── queue/               # Priority queue
│   │   └── storage/             # File system storage
│   └── pkg/
│       ├── config/              # Configuration
│       └── models/              # Data models
├── frontend/
│   ├── src/
│   │   ├── components/          # React components (atoms/molecules/organisms)
│   │   ├── pages/               # Route pages
│   │   ├── hooks/               # Custom hooks
│   │   └── api/                 # API client
│   └── vite.config.ts
├── dev-all.sh                   # Start full stack
├── stop-dev.sh                  # Stop all services
└── Makefile                     # Build automation

License

Private