README

HTTP Traffic Capture System with mitmproxy

A Docker-based HTTP traffic interception and monitoring system that captures both incoming and outgoing HTTP/HTTPS traffic using mitmproxy. The system provides real-time visibility into API calls with full request/response data through a web interface.

📚 Documentation

For comprehensive integration guides, examples, and advanced configuration:

👉 View Complete Documentation

Features

Complete HTTP/HTTPS Traffic Capture: Captures both incoming client requests and outgoing API calls
Hierarchical Request Organization: Groups internal API calls under their parent client requests
Real-Time Web UI: Live monitoring interface at http://localhost:8090
Full Request/Response Bodies: Unlike packet capture, sees complete HTTP data including headers and bodies
Trace ID Correlation: Automatically tracks requests through the system with unique trace IDs
WebSocket Updates: Real-time flow updates without page refresh
JSON Formatting: Automatic pretty-printing of JSON payloads
Scroll Pinning: Selected flows stay in place when new traffic arrives
Smooth Animations: Enhanced UI with scroll animations and transitions

Architecture

The system uses mitmproxy in dual-proxy mode to capture all HTTP traffic:

mitmproxy: HTTP/HTTPS intercepting proxy running in both reverse and forward proxy modes

- Reverse proxy (port 8080): Captures incoming client requests - Forward proxy (port 8082): Captures outgoing API calls from rest-caller - Web UI (port 8090): Real-time traffic monitoring interface

rest-caller: REST API service that accepts requests and forwards them to external APIs

- Configured to use mitmproxy as forward proxy for all outgoing requests - Adds trace IDs for request correlation

Shared Network Namespace: rest-caller and mitmproxy share the same network namespace for seamless traffic capture

Quick Start

bash

# Start all services
docker-compose up -d

# Wait for services to start
sleep 5

# Test with browser-friendly endpoints
# Visit these URLs in your browser:
# - http://localhost:8080/demo  (simple GET request)
# - http://localhost:8080/test  (POST request with data)

# View live traffic monitor
open http://localhost:8090

# View service logs
docker-compose logs -f rest-caller
docker-compose logs -f mitmproxy

API Endpoints

Browser-Friendly Test Endpoints

#### GET /demo Simple endpoint that makes an external API call:

bash

curl http://localhost:8080/demo
# Or visit in browser: http://localhost:8080/demo

#### GET /test Test endpoint that makes a POST request with JSON data:

bash

curl http://localhost:8080/test
# Or visit in browser: http://localhost:8080/test

Main API Endpoint

#### POST /fetch Make outbound HTTP/HTTPS requests through the forwarder:

bash

curl -X POST 'http://localhost:8080/fetch?maxChars=500' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://api.example.com/resource",
    "method": "GET",
    "headers": {"Accept": "application/json"},
    "body": "",
    "body_mode": "raw"
  }'

Request Parameters:

url (required): Target URL to fetch
method (required): HTTP method (GET, POST, PUT, DELETE, PATCH, HEAD)
headers (optional): Key-value pairs of HTTP headers
body (optional): Request body content
body_mode (optional): "raw" or "base64" for body encoding

Query Parameters:

maxChars: Limit response body to N characters (default: 2000)

Response Headers:

X-Trace-Id: Unique request identifier for correlation
X-Upstream-Status: Original status code from upstream
X-Truncated: "true" if response was truncated
X-Original-Body-Length: Original size before truncation

Traffic Monitor Web UI

Access the live traffic monitor at http://localhost:8090

UI Screenshot

[](docs/mitmproxy-ui-large.png) Click image to view full resolution (2880x1800)

The web interface provides real-time monitoring of all HTTP traffic flowing through the system, with:

Total request statistics showing client calls and internal API calls
Hierarchical flow organization with expandable request details
Response status codes and timing information for each request
Client/Internal call differentiation with clear visual tags
One-click flow clearing to reset the captured traffic

The monitor displays captured HTTP traffic in a hierarchical structure:

Flow Organization

Client Requests: Top-level requests from external clients (browser, curl, etc.)
Internal Calls: API calls made by rest-caller, grouped under their parent client request
Trace ID Correlation: Automatic grouping using X-Trace-Id headers

Flow Details

Each captured flow shows:

Timestamp: When the request was made
Method: HTTP method (GET, POST, etc.)
URL: Complete request URL
Status: HTTP response status code
Duration: Total request time in milliseconds
Headers: All request and response headers
Body: Complete request and response bodies (formatted for JSON)

UI Features

Real-time Updates: New flows appear automatically via WebSocket
Clear Flows: Button to clear all captured traffic
Expandable Details: Click flows to see full request/response data
JSON Formatting: Automatic pretty-printing of JSON payloads
Copy Support: Easy copying of request/response data

Examples

Simple GET Request

bash

curl -X POST 'http://localhost:8080/fetch' \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://httpbin.org/json","method":"GET"}'

POST with JSON Body

bash

curl -X POST 'http://localhost:8080/fetch' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://httpbin.org/post",
    "method": "POST",
    "headers": {"Content-Type": "application/json"},
    "body": "{\"hello\":\"world\"}",
    "body_mode": "raw"
  }'

Base64 Encoded Body

bash

ENCODED=$(echo -n '{"data":"binary content"}' | base64)
curl -X POST 'http://localhost:8080/fetch' \
  -H 'Content-Type: application/json' \
  -d "{
    \"url\": \"https://httpbin.org/post\",
    \"method\": \"POST\",
    \"body\": \"$ENCODED\",
    \"body_mode\": \"base64\"
  }"

How It Works

Traffic Capture Flow

Client Request → mitmproxy (reverse proxy on 8080) → rest-caller (8081)

rest-caller generates a unique X-Trace-Id header

Outgoing API Call → mitmproxy (forward proxy on 8082) → External API

mitmproxy captures both requests with the same trace ID

Web UI groups the calls hierarchically by trace ID

Proxy Configuration

The system uses a programmatic proxy configuration in rest-caller:

// Configure transport with mitmproxy
proxyURL, _ := url.Parse("http://localhost:8082")
transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
    TLSClientConfig: &tls.Config{
        InsecureSkipVerify: true, // Required for proxy interception
    },
}

This approach avoids using HTTP_PROXY environment variables and ensures all outgoing traffic goes through mitmproxy.

Configuration

Environment Variables

rest-caller:

PORT: Service port (default: 8081)
DEFAULT_MAX_CHARS: Default response truncation (default: 2000)
UPSTREAM_TIMEOUT: Request timeout (default: 10s)
MAX_READ_BYTES: Maximum response size (default: 10MB)
LOG_LEVEL: info or debug (default: info)

mitmproxy:

MITMPROXY_OPTIONS: Additional mitmproxy options

Docker Compose Networking

The system uses Docker's shared network namespace feature:

yaml

services:
  rest-caller:
    network_mode: "service:mitmproxy"  # Share network with mitmproxy
  
  mitmproxy:
    ports:
      - "8080:8080"  # Reverse proxy
      - "8082:8082"  # Forward proxy
      - "8090:8090"  # Web UI

This configuration allows both services to share the same network stack, enabling seamless traffic capture.

Troubleshooting

No Traffic Visible in UI

Check mitmproxy is running:

bash

   docker-compose ps
   docker-compose logs mitmproxy

Verify services are sharing network namespace:

bash

   docker inspect docker_rest-caller_1 | grep NetworkMode
   # Should show: "NetworkMode": "container:docker_mitmproxy_1"

Test with demo endpoint:

bash

   curl http://localhost:8080/demo
   # Check UI at http://localhost:8090

Connection Refused Errors

Ensure all services are up:

bash

   docker-compose up -d
   docker-compose ps

Check port availability:

bash

   lsof -i :8080,8082,8090

Restart services:

bash

   docker-compose restart

Flows Not Grouped Properly

Ensure rest-caller is adding X-Trace-Id headers
Check trace ID is being propagated in outgoing requests
View raw flow details in UI to verify trace IDs match

Development

Project Structure

shell

.
├── docker-compose.yml           # Service orchestration
├── Dockerfile.mitmproxy        # mitmproxy container build
├── Dockerfile.restcaller       # rest-caller container build
├── mitmproxy/
│   ├── unified.py              # Main mitmproxy addon with web server
│   ├── index.html              # Web UI for traffic monitoring
│   └── forward-entrypoint.sh   # Startup script for dual proxy modes
└── rest-caller/
    ├── main.go                 # REST API service with proxy config
    ├── go.mod                  # Go module definition
    └── go.sum                  # Go dependencies

Building Locally

bash

# Build all services
docker-compose build

# Build specific service
docker-compose build mitmproxy
docker-compose build rest-caller

Running Without Docker

bash

# Terminal 1: Start mitmproxy
cd mitmproxy
mitmdump -s unified.py \
  --mode reverse:http://localhost:8081@8080 \
  --mode regular@8082

# Terminal 2: Start rest-caller
cd rest-caller
PORT=8081 go run main.go

# Terminal 3: Access web UI
open http://localhost:8090

Advanced Features

Custom mitmproxy Addons

The unified.py addon can be extended with additional functionality:

Request/response modification
Custom filtering logic
Additional metadata extraction
Integration with external systems

Trace ID Propagation

The system automatically propagates X-Trace-Id headers through all requests. This enables:

Request correlation across microservices
Distributed tracing integration
Performance monitoring
Debugging complex request flows

Security Notes

TLS Interception: mitmproxy decrypts HTTPS traffic for inspection
InsecureSkipVerify: Required for proxy to intercept TLS connections
Local Development Only: This setup is designed for development/debugging
Sensitive Data: Be aware that all request/response data is captured
Network Isolation: Consider running in isolated network for production testing

About

This is an experimental HTTP traffic monitoring system coded using Claude Code. It's designed for development exploration and learning - not intended for reuse in production environments. The project demonstrates modern web UI patterns, real-time WebSocket communication, and containerized service orchestration.

🚀 Looking for a free weekend to try this out and see if it's a solution to check the traffic flowing into and out of a standalone app!

🧪 This is an experimental project built for fun and learning - not intended as production software!