Back to Robin AI Assistant

README

Robin - Smart Home Companion for the Waveshare ESP32-S3-RLCD-4.2

Robin is a voice-activated personal assistant and smart home companion built for the Waveshare ESP32-S3-RLCD-4.2 reflective LCD board. It combines a 400x300 1-bit reflective display, Zigbee home sensor monitoring, voice interaction via hold-to-talk, and an LLM-powered conversational assistant - all designed to help a home-bound individual stay connected, safe, and supported.

What It Does

  • Voice Assistant ("Robin") - Hold the KEY button to speak. Robin transcribes speech (Whisper), thinks (Ollama/Qwen2.5), and speaks back (Piper TTS) through the onboard speaker. Robin can search the web, manage a shopping list, set reminders, save notes to memory, and send push notifications to family.
  • Home Monitoring - 9 Zigbee sensors (door contacts, presence detectors, vibration sensor, button, temperature/humidity, plant soil moisture) stream real-time data via WebSocket from a GraphQL backend. The dashboard shows all devices at a glance.
  • Safety Escalation - If vibration (possible fall) or a button press is detected, Robin speaks to check on the user. If no response after 30s, 90s, and 3 minutes, it escalates through spoken alerts to push notifications to family via ntfy.
  • 40+ Display Screens - Cycle through screens with BOOT (next) and KEY (previous) buttons. Includes home dashboard, conversation view, productivity tools (kanban, habit tracker, shopping list, weekly planner), information displays (weather, clocks, birthdays), and visual animations (Mandelbrot, reaction diffusion, maze generation, Pong, and more).
  • Daily Wellness - Morning, midday, and evening check-ins. 24-hour inactivity triggers a wellness alert to family.

Architecture Overview

shell
                    +-----------------------+
                    |   Zigbee Sensors (9)  |
                    |  door, presence, etc  |
                    +-----------+-----------+
                                |
                         zigbee2mqtt + MQTT
                                |
                    +-----------v-----------+
                    |   GraphQL Backend     |
                    |   (gqlgen, Go)        |
                    |   :8080               |
                    +-----------+-----------+
                                |
              WebSocket (real-time events)
              HTTP POST (initial poll)
                                |
+---------------+   +-----------v-----------+   +------------------+
|  ESP32-S3     |   |   KickPi Server       |   |  AI Services     |
|  RLCD-4.2     |<->|   (audio-receiver)    |<->|  (GPU Server)    |
|               |   |   :8081               |   |                  |
|  - 400x300    |   |                       |   |  Whisper ASR     |
|    1-bit LCD  |   |  - Voice pipeline     |   |  Ollama LLM      |
|  - ES8311 DAC |   |  - LLM tool calling   |   |  Piper TTS       |
|  - ES7210 ADC |   |  - Escalation engine  |   |                  |
|  - WiFi       |   |  - Family dashboard   |   +------------------+
|  - Speaker    |   |  - Reminder system    |
|  - Microphone |   |  - Push notifications |
+---------------+   +-----------------------+

See docs/ARCHITECTURE.md for detailed component documentation.

Hardware

  • Board: Waveshare ESP32-S3-RLCD-4.2
  • Display: 400x300 reflective LCD (ST7305 driver) - paper-like, readable in sunlight, no backlight needed
  • Audio: ES8311 DAC (speaker) + ES7210 ADC (dual microphone)
  • Connectivity: WiFi 802.11 b/g/n, 16MB flash, 8MB PSRAM (octal)
  • Buttons: BOOT (next screen), KEY (short=prev screen, hold=voice record)

Quick Start

Prerequisites

  • ESP-IDF v5.5+
  • Go 1.24+ (for audio-receiver backend)
  • Piper TTS, Whisper ASR, and Ollama running on a GPU server
  • Zigbee2MQTT + GraphQL backend (separate project)

Build & Flash ESP32 Firmware

bash
cd animation-demo
source ~/esp/esp-idf/export.sh
idf.py build
idf.py -p /dev/ttyACM0 flash monitor

Deploy Audio Receiver (KickPi / ARM64)

bash
cd audio-receiver
GOOS=linux GOARCH=arm64 go build -o audio-receiver-arm64
scp audio-receiver-arm64 kickpi@192.168.0.164:/home/kickpi/audio-receiver/audio-receiver
ssh kickpi@192.168.0.164 'nohup /home/kickpi/audio-receiver/audio-receiver > /tmp/audio-receiver.log 2>&1 &'

Start TTS Server

bash
cd audio-receiver
python3 tts-server.py  # Runs on port 9001

Documentation

Project Structure

shell
epaper-esp32/
├── animation-demo/          # ESP32 firmware (ESP-IDF C++)
│   ├── main/
│   │   ├── main.cpp         # Entry point, button handling, animation loop
│   │   ├── graphics.h       # Display primitives (pixel, line, rect, circle, dither)
│   │   ├── font5x7.h        # Bitmap font for text rendering
│   │   ├── wifi_manager.h   # WiFi STA connection
│   │   ├── audio_alert.h    # I2S audio, codec init, recording, TTS playback server
│   │   ├── zigbee_poller.h  # WebSocket subscription to Zigbee backend
│   │   ├── page_conversation.h      # Robin voice assistant UI
│   │   ├── page_zigbee_dashboard.h  # Sensor overview dashboard
│   │   ├── anim_*.h         # Visual animation screens (10)
│   │   ├── todo_*.h         # Productivity/task screens (11)
│   │   └── page_*.h         # Information/utility screens (12)
│   ├── components/
│   │   └── display_driver/  # ST7305 reflective LCD driver
│   ├── sdkconfig.defaults   # ESP32 build config
│   └── partitions.csv       # Flash partition layout
│
├── audio-receiver/          # Go backend server (runs on KickPi)
│   ├── main.go              # HTTP server, LLM integration, escalation engine
│   ├── tts-server.py        # Piper TTS HTTP wrapper
│   ├── go.mod               # Go module definition
│   └── piper-voices/        # TTS voice model files
│       └── en_GB-alba-medium.onnx.json
│
└── docs/                    # Documentation
    ├── ARCHITECTURE.md
    ├── USER_GUIDE.md
    ├── DEVELOPER_GUIDE.md
    └── API_REFERENCE.md

License

Private project. All rights reserved.

© 2026 Jonathan Leahy · v0.9.1