README

Web Spider 🕷️

A high-performance, concurrent web crawler built with Go and TypeScript. Features intelligent caching, robots.txt compliance, and beautiful Markdown conversion.

Features

Concurrent Crawling: Configurable worker pool for parallel page processing
Smart Caching: Time-based cache with configurable expiration
Resume Capability: Persistent queue allows stopping and resuming crawls
Depth-First Crawling: Automatically detects and respects maximum depth
HTML to Markdown: Converts crawled pages to clean Markdown format
Robots.txt Compliance: Respects website crawling rules
Rate Limiting: Polite crawling with configurable delays
Modern UI: Real-time dashboard with Tailwind CSS and shadcn styling
REST API: Full control via HTTP endpoints

Architecture

shell

web-spider/
├── backend/           # Go crawler engine
│   ├── cmd/          # Application entry point
│   ├── internal/     # Core business logic
│   │   ├── api/      # REST API handlers
│   │   ├── cache/    # Caching system
│   │   ├── converter/# HTML to Markdown
│   │   ├── crawler/  # Main crawler engine
│   │   ├── queue/    # Priority queue
│   │   └── storage/  # File system storage
│   └── pkg/          # Shared packages
├── frontend/         # TypeScript UI
│   └── src/
│       ├── api/      # API client
│       └── components/# UI components
└── data/            # Crawled data storage
    ├── pages/       # Markdown pages
    ├── links/       # Link metadata
    └── cache/       # Cache storage

Quick Start

Prerequisites

Go 1.21+
Node.js 18+
Make (optional)

Installation

bash

# Clone the repository
git clone https://github.com/yourusername/web-spider.git
cd web-spider

# Install dependencies
make install-deps

# Or manually:
cd backend && go mod download
cd ../frontend && npm install

Running

bash

# Production build
make build
make run

# Development mode (with hot reload)
make dev

# Or run separately:
make backend-dev  # Terminal 1
make frontend-dev # Terminal 2

The application will be available at:

Frontend: http://localhost:3001
API: http://localhost:8090

Configuration

Default Settings

json

{
  "crawler": {
    "max_workers": 5,
    "max_depth": 10,
    "cache_duration": "24h",
    "rate_limit": "1s",
    "respect_robots": true,
    "user_agent": "WebSpider/1.0"
  }
}

Environment Variables

bash

SPIDER_PORT=8090          # API server port
SPIDER_HOST=0.0.0.0      # API server host
SPIDER_DATA_DIR=./data   # Data storage directory

API Endpoints

Crawler Control

bash

# Start crawling
POST /api/v1/crawl/start
{
  "url": "https://example.com"
}

# Stop crawling
POST /api/v1/crawl/stop

# Get statistics
GET /api/v1/crawl/stats

# Get/Update configuration
GET /api/v1/crawl/config
PUT /api/v1/crawl/config

Data Access

bash

# List crawled domains
GET /api/v1/domains

# Get cached pages
GET /api/v1/pages/{domain}

# Get extracted links
GET /api/v1/links/{domain}

# Cache management
GET /api/v1/cache/stats
DELETE /api/v1/cache

# Queue management
GET /api/v1/queue
DELETE /api/v1/queue

Storage Structure

shell

data/
├── pages/
│   └── example.com/
│       ├── index.md           # Converted markdown
│       └── index.md.meta.json # Metadata
├── links/
│   └── example.com/
│       └── [hash]_links.json  # Extracted links
├── queue.json                  # Persistent queue
└── stats.json                  # Crawl statistics

Advanced Usage

Custom Configuration File

bash

# Create config.json
{
  "server": {
    "port": "8090",
    "host": "0.0.0.0"
  },
  "crawler": {
    "max_workers": 10,
    "max_depth": 5,
    "cache_duration": "12h",
    "rate_limit": "500ms"
  }
}

# Run with custom config
./bin/web-spider -config config.json

Docker Deployment

bash

# Build image
docker build -t web-spider:latest .

# Run container
docker run -d \
  -p 8090:8090 \
  -p 3001:3001 \
  -v $(pwd)/data:/app/data \
  web-spider:latest

Development

Testing

bash

# Run all tests
make test

# Run specific package tests
cd backend && go test ./internal/crawler -v

Building from Source

bash

# Backend only
cd backend
go build -o ../bin/web-spider cmd/main.go

# Frontend only
cd frontend
npm run build

Features in Detail

Concurrent Worker Pool

Dynamic worker scaling
Graceful shutdown
Automatic retry on failure

Intelligent Caching

URL-based cache keys
Configurable TTL
Automatic cache invalidation

Resume Capability

Persistent queue state
Automatic recovery on restart
Progress tracking

Depth Detection

Automatic max depth discovery
Priority-based crawling
Breadth-first option available

Troubleshooting

Common Issues

Port already in use

bash

   # Change ports in config or environment
   SPIDER_PORT=8081 ./bin/web-spider

Permission denied for data directory

bash

   # Fix permissions
   chmod -R 755 ./data

Out of memory with large sites

bash

   # Reduce workers and increase rate limit
   # Adjust in UI settings or config file

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details

Acknowledgments

Built with Go and TypeScript
UI styled with Tailwind CSS
HTML parsing with goquery
Markdown conversion with custom algorithms