Skip to content

Emergency Break Glass Protocol - Configuration Guide

Version: 1.0 Last Updated: January 26, 2026 Purpose: Complete reference for configuring emergency break glass access


Table of Contents


Overview

Charon's emergency break glass protocol provides a 3-tier system for emergency access recovery:

  • Tier 1: Emergency token via main application endpoint (Layer 7 bypass)
  • Tier 2: Separate emergency server on dedicated port (network isolation)
  • Tier 3: Direct system access (SSH/console)

This guide covers configuration for Tiers 1 and 2. Tier 3 requires only SSH access to the host.


Environment Variables Reference

Required Variables

CHARON_EMERGENCY_TOKEN

Purpose: Secret token for emergency break glass access (Tier 1 & 2) Format: 64-character hexadecimal string Security: CRITICAL - Store in secrets manager, never commit to version control

Generation:

# Recommended method (OpenSSL)
openssl rand -hex 32

# Alternative (Python)
python3 -c "import secrets; print(secrets.token_hex(32))"

# Alternative (/dev/urandom)
head -c 32 /dev/urandom | xxd -p -c 64

Example (generate your own token with openssl rand -hex 32; never reuse an example value):

environment:
  - CHARON_EMERGENCY_TOKEN=<YOUR_EMERGENCY_TOKEN>

Validation:

  • Minimum length: 32 characters (produces 64-char hex)
  • Must be hexadecimal (0-9, a-f)
  • Must be unique per deployment
  • Rotate every 90 days

Optional Variables

CHARON_MANAGEMENT_CIDRS

Purpose: IP ranges allowed to use emergency token (Tier 1) Format: Comma-separated CIDR notation Default: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8 (RFC1918 + localhost)

Examples:

# Office network only
- CHARON_MANAGEMENT_CIDRS=192.168.1.0/24

# Office + VPN
- CHARON_MANAGEMENT_CIDRS=192.168.1.0/24,10.8.0.0/24

# Multiple offices
- CHARON_MANAGEMENT_CIDRS=192.168.1.0/24,192.168.2.0/24,10.10.0.0/16

# Allow from anywhere (NOT RECOMMENDED)
- CHARON_MANAGEMENT_CIDRS=0.0.0.0/0,::/0

The emergency server uses this same list, with the same defaults as the main Charon API (private ranges plus localhost). If you connect from anywhere else, such as Tailscale (100.64.0.0/10) or a public VPN range, add that range here or the emergency server will refuse you.

You can list either a network range (like 192.168.1.0/24) or a single address on its own (like 10.0.0.5). A single address is treated as just that one computer.

If an entry is typed wrong: Charon writes an error to its log that names the bad entry. Until you fix it, emergency access works only from the Charon machine itself. Fix the entry and restart Charon to restore access from your other networks.

Security Notes:

  • Be as restrictive as possible
  • Never use 0.0.0.0/0 in production
  • Include VPN subnet if using VPN for emergency access
  • Update when office networks change

CHARON_EMERGENCY_SERVER_ENABLED

Purpose: Enable separate emergency server on dedicated port (Tier 2) Format: Boolean (true or false) Default: false

When to enable:

  • ✅ Production deployments with CrowdSec
  • ✅ High-security environments
  • ✅ Deployments with restrictive firewalls
  • ❌ Simple home labs (Tier 1 sufficient)

Example:

environment:
  - CHARON_EMERGENCY_SERVER_ENABLED=true

CHARON_EMERGENCY_BIND

Purpose: Address and port for emergency server (Tier 2) Format: IP:PORT Default: 127.0.0.1:2020 Note: Port 2020 is the emergency server. Caddy's admin API is a unix socket inside the container and uses no TCP port.

While the emergency server is turned on, its port cannot be used as a proxy target or an uptime monitor address on this machine (localhost, 127.0.0.1 or the machine's own address). Charon refuses those entries. Turn the emergency server off, or use another port for proxy targets and monitors.

Options:

# Localhost only (most secure - requires SSH tunnel)
- CHARON_EMERGENCY_BIND=127.0.0.1:2020

# Listen on all interfaces (DANGER - requires firewall rules)
- CHARON_EMERGENCY_BIND=0.0.0.0:2020

# Specific internal IP (VPN interface)
- CHARON_EMERGENCY_BIND=10.8.0.1:2020

# IPv6 localhost
- CHARON_EMERGENCY_BIND=[::1]:2020

# Dual-stack all interfaces
- CHARON_EMERGENCY_BIND=0.0.0.0:2020  # or [::]:2020 for IPv6

⚠️ Security Warning: Never bind to 0.0.0.0 without firewall protection. Use SSH tunneling instead.

When Charon starts, it writes a warning to its log if the emergency server listens on an address other than loopback (127.0.0.1 or [::1]). The emergency server serves plain HTTP, so only expose it through a VPN or a tunnel you trust. The warning is a reminder: Charon still starts.

CHARON_EMERGENCY_USERNAME

Purpose: Basic Auth username for emergency server (Tier 2) Format: String Default: None (Basic Auth disabled)

Example:

environment:
  - CHARON_EMERGENCY_USERNAME=admin

Security Notes:

  • Optional but recommended
  • Use strong, unique username (not "admin" in production)
  • Combine with strong password
  • Consider using mTLS instead (future enhancement)

CHARON_EMERGENCY_PASSWORD

Purpose: Basic Auth password for emergency server (Tier 2) Format: String Default: None (Basic Auth disabled)

Example:

environment:
  - CHARON_EMERGENCY_PASSWORD=${EMERGENCY_PASSWORD}  # From .env file

Security Notes:

  • NEVER hardcode in docker-compose.yml
  • Use .env file or secrets manager
  • Minimum 20 characters recommended
  • Rotate every 90 days

Reset Attempt Limits

To stop anyone from guessing your emergency token, Charon limits wrong attempts for each address:

  • Each address gets 5 failed attempts in a 5-minute window. A failed attempt is one with a wrong or missing token. Requests refused for other reasons, such as coming from outside the management network, do not count.
  • After that, each extra failed attempt needs a wait of about one minute before the next try. Charon answers "Too many requests" until the wait is over.
  • The main Charon port and the local-only emergency port (Tier 2) keep separate counters, so attempts on one never use up the other.
  • A successful reset does not use up any attempts. However, if your address is already waiting out a limit, you must wait for the timer to finish before trying again, even with the correct token.

Reset Reply Time

A reset can take a while to finish. Charon may first wait for another change to finish applying, then applies the reset and sends the reply. The emergency server allows up to about 75 seconds for this, so you get the reply instead of a dropped connection. If you don't get a reply, wait for the page or command to finish before trying again.

A reset sent through a tunnel on the same machine is judged by the visitor's address, not by 127.0.0.1. A reset sent directly to the local port is not affected. See Same-Machine Tunnels.


Docker Compose Examples

Example 1: Minimal Configuration (Homelab)

Use case: Simple home lab, Tier 1 only, no emergency server

version: '3.8'

services:
  charon:
    image: ghcr.io/wikid82/charon:latest
    container_name: charon
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
      - "8080:8080"
    volumes:
      - charon_data:/app/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TZ=UTC
      - CHARON_ENV=production
      - CHARON_ENCRYPTION_KEY=${CHARON_ENCRYPTION_KEY}  # From .env
      - CHARON_EMERGENCY_TOKEN=${CHARON_EMERGENCY_TOKEN}  # From .env

volumes:
  charon_data:
    driver: local

.env file:

# Generate with: openssl rand -base64 32
CHARON_ENCRYPTION_KEY=your-32-byte-base64-key-here

# Generate with: openssl rand -hex 32
CHARON_EMERGENCY_TOKEN=your-64-char-hex-token-here

Example 2: Production Configuration (Tier 1 + Tier 2)

Use case: Production deployment with emergency server, VPN access

version: '3.8'

services:
  charon:
    image: ghcr.io/wikid82/charon:latest
    container_name: charon
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
      - "8080:8080"
      # Emergency server (localhost only - use SSH tunnel)
      - "127.0.0.1:2020:2020"
    volumes:
      - charon_data:/app/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TZ=UTC
      - CHARON_ENV=production
      - CHARON_ENCRYPTION_KEY=${CHARON_ENCRYPTION_KEY}

      # Emergency Token (Tier 1)
      - CHARON_EMERGENCY_TOKEN=${CHARON_EMERGENCY_TOKEN}
      - CHARON_MANAGEMENT_CIDRS=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

      # Emergency Server (Tier 2)
      - CHARON_EMERGENCY_SERVER_ENABLED=true
      - CHARON_EMERGENCY_BIND=0.0.0.0:2020
      - CHARON_EMERGENCY_USERNAME=${CHARON_EMERGENCY_USERNAME}
      - CHARON_EMERGENCY_PASSWORD=${CHARON_EMERGENCY_PASSWORD}
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v1/health"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  charon_data:
    driver: local

.env file:

CHARON_ENCRYPTION_KEY=your-32-byte-base64-key-here
CHARON_EMERGENCY_TOKEN=your-64-char-hex-token-here
CHARON_EMERGENCY_USERNAME=emergency-admin
CHARON_EMERGENCY_PASSWORD=your-strong-password-here

Example 3: Security-Hardened Configuration

Use case: High-security environment with Docker secrets, read-only filesystem

version: '3.8'

services:
  charon:
    image: ghcr.io/wikid82/charon:latest
    container_name: charon
    restart: unless-stopped
    read_only: true
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    security_opt:
      - no-new-privileges:true
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
      - "8080:8080"
      - "127.0.0.1:2020:2020"
    volumes:
      - charon_data:/app/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # tmpfs for writable directories
      - type: tmpfs
        target: /tmp
        tmpfs:
          size: 100M
      - type: tmpfs
        target: /var/log/caddy
        tmpfs:
          size: 100M
    secrets:
      - charon_encryption_key
      - charon_emergency_token
      - charon_emergency_password
    environment:
      - TZ=UTC
      - CHARON_ENV=production
      - CHARON_ENCRYPTION_KEY_FILE=/run/secrets/charon_encryption_key
      - CHARON_EMERGENCY_TOKEN_FILE=/run/secrets/charon_emergency_token
      - CHARON_MANAGEMENT_CIDRS=10.8.0.0/24  # VPN subnet only
      - CHARON_EMERGENCY_SERVER_ENABLED=true
      - CHARON_EMERGENCY_BIND=0.0.0.0:2020
      - CHARON_EMERGENCY_USERNAME=emergency-admin
      - CHARON_EMERGENCY_PASSWORD_FILE=/run/secrets/charon_emergency_password

volumes:
  charon_data:
    driver: local

secrets:
  charon_encryption_key:
    external: true
  charon_emergency_token:
    external: true
  charon_emergency_password:
    external: true

Create secrets:

# Create secrets from files
echo "your-encryption-key" | docker secret create charon_encryption_key -
echo "your-emergency-token" | docker secret create charon_emergency_token -
echo "your-emergency-password" | docker secret create charon_emergency_password -

# Verify secrets
docker secret ls

Example 4: Development Configuration

Use case: Local development, emergency server for testing

version: '3.8'

services:
  charon:
    image: ghcr.io/wikid82/charon:nightly
    container_name: charon-dev
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
      - "2020:2020"  # Emergency server on all interfaces for testing
    volumes:
      - charon_data:/app/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TZ=UTC
      - CHARON_ENV=development
      - CHARON_DEBUG=1
      - CHARON_ENCRYPTION_KEY=dev-key-not-for-production-32bytes
      - CHARON_EMERGENCY_TOKEN=test-emergency-token-for-e2e-32chars
      - CHARON_EMERGENCY_SERVER_ENABLED=true
      - CHARON_EMERGENCY_BIND=0.0.0.0:2020
      - CHARON_EMERGENCY_USERNAME=admin
      - CHARON_EMERGENCY_PASSWORD=admin

volumes:
  charon_data:
    driver: local

⚠️ WARNING: This configuration is ONLY for local development. Never use in production.


Firewall Configuration

iptables Rules (Linux)

Block public access to emergency port:

# Allow localhost
iptables -A INPUT -i lo -p tcp --dport 2020 -j ACCEPT

# Allow VPN subnet (example: 10.8.0.0/24)
iptables -A INPUT -s 10.8.0.0/24 -p tcp --dport 2020 -j ACCEPT

# Block everything else
iptables -A INPUT -p tcp --dport 2020 -j DROP

# Save rules
iptables-save > /etc/iptables/rules.v4

UFW Rules (Ubuntu/Debian)

# Allow from specific subnet only
ufw allow from 10.8.0.0/24 to any port 2020 proto tcp

# Enable firewall
ufw enable

# Verify rules
ufw status numbered

firewalld Rules (RHEL/CentOS)

# Create new zone for emergency access
firewall-cmd --permanent --new-zone=emergency
firewall-cmd --permanent --zone=emergency --add-source=10.8.0.0/24
firewall-cmd --permanent --zone=emergency --add-port=2020/tcp

# Reload firewall
firewall-cmd --reload

# Verify
firewall-cmd --zone=emergency --list-all

Docker Network Isolation

Create dedicated network for emergency access:

services:
  charon:
    networks:
      - public
      - emergency

networks:
  public:
    driver: bridge
  emergency:
    driver: bridge
    internal: true  # No external connectivity

Secrets Manager Integration

HashiCorp Vault

Store secrets:

# Store emergency token
vault kv put secret/charon/emergency \
  token="$(openssl rand -hex 32)" \
  username="emergency-admin" \
  password="$(openssl rand -base64 32)"

# Read secrets
vault kv get secret/charon/emergency

Docker Compose with Vault:

services:
  charon:
    image: ghcr.io/wikid82/charon:latest
    environment:
      - CHARON_EMERGENCY_TOKEN=${VAULT_CHARON_EMERGENCY_TOKEN}
      - CHARON_EMERGENCY_USERNAME=${VAULT_CHARON_EMERGENCY_USERNAME}
      - CHARON_EMERGENCY_PASSWORD=${VAULT_CHARON_EMERGENCY_PASSWORD}

Retrieve from Vault:

# Export secrets from Vault
export VAULT_CHARON_EMERGENCY_TOKEN=$(vault kv get -field=token secret/charon/emergency)
export VAULT_CHARON_EMERGENCY_USERNAME=$(vault kv get -field=username secret/charon/emergency)
export VAULT_CHARON_EMERGENCY_PASSWORD=$(vault kv get -field=password secret/charon/emergency)

# Start with secrets
docker-compose up -d

AWS Secrets Manager

Store secrets:

# Create secret
aws secretsmanager create-secret \
  --name charon/emergency \
  --description "Charon emergency break glass credentials" \
  --secret-string '{
    "token": "YOUR_TOKEN_HERE",
    "username": "emergency-admin",
    "password": "YOUR_PASSWORD_HERE"
  }'

Retrieve in Docker Compose:

#!/bin/bash

# Retrieve secret
SECRET=$(aws secretsmanager get-secret-value \
  --secret-id charon/emergency \
  --query SecretString \
  --output text)

# Parse JSON and export
export CHARON_EMERGENCY_TOKEN=$(echo $SECRET | jq -r '.token')
export CHARON_EMERGENCY_USERNAME=$(echo $SECRET | jq -r '.username')
export CHARON_EMERGENCY_PASSWORD=$(echo $SECRET | jq -r '.password')

# Start Charon
docker-compose up -d

Azure Key Vault

Store secrets:

# Create Key Vault
az keyvault create \
  --name charon-vault \
  --resource-group charon-rg \
  --location eastus

# Store secrets
az keyvault secret set \
  --vault-name charon-vault \
  --name emergency-token \
  --value "YOUR_TOKEN_HERE"

az keyvault secret set \
  --vault-name charon-vault \
  --name emergency-username \
  --value "emergency-admin"

az keyvault secret set \
  --vault-name charon-vault \
  --name emergency-password \
  --value "YOUR_PASSWORD_HERE"

Retrieve secrets:

#!/bin/bash

# Retrieve secrets
export CHARON_EMERGENCY_TOKEN=$(az keyvault secret show \
  --vault-name charon-vault \
  --name emergency-token \
  --query value -o tsv)

export CHARON_EMERGENCY_USERNAME=$(az keyvault secret show \
  --vault-name charon-vault \
  --name emergency-username \
  --query value -o tsv)

export CHARON_EMERGENCY_PASSWORD=$(az keyvault secret show \
  --vault-name charon-vault \
  --name emergency-password \
  --query value -o tsv)

# Start Charon
docker-compose up -d

Security Hardening

Best Practices Checklist

  • [ ] Emergency token stored in secrets manager (not in docker-compose.yml)
  • [ ] Token rotation scheduled every 90 days
  • [ ] Management CIDRs restricted to minimum necessary networks
  • [ ] Emergency server bound to localhost only (127.0.0.1)
  • [ ] SSH tunneling used for emergency server access
  • [ ] Firewall rules block public access to port 2020
  • [ ] Basic Auth enabled on emergency server with strong credentials
  • [ ] Audit logging monitored for emergency access
  • [ ] Alerts configured for emergency token usage
  • [ ] Backup procedures tested and documented
  • [ ] Recovery runbooks reviewed by team
  • [ ] Quarterly drills scheduled to test procedures

Network Hardening

VPN-Only Access:

environment:
  # Only allow emergency access from VPN subnet
  - CHARON_MANAGEMENT_CIDRS=10.8.0.0/24

  # Emergency server listens on VPN interface only
  - CHARON_EMERGENCY_BIND=10.8.0.1:2020

mTLS for Emergency Server (Future Enhancement):

environment:
  - CHARON_EMERGENCY_TLS_ENABLED=true
  - CHARON_EMERGENCY_TLS_CERT=/run/secrets/emergency_tls_cert
  - CHARON_EMERGENCY_TLS_KEY=/run/secrets/emergency_tls_key
  - CHARON_EMERGENCY_TLS_CA=/run/secrets/emergency_tls_ca

Monitoring & Alerting

Prometheus Metrics:

# Emergency access metrics
charon_emergency_token_attempts_total{result="success"}
charon_emergency_token_attempts_total{result="failure"}
charon_emergency_server_requests_total

Alert Rules:

groups:
  - name: charon_emergency_access
    rules:
      - alert: EmergencyTokenUsed
        expr: increase(charon_emergency_token_attempts_total{result="success"}[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Emergency break glass token was used"
          description: "Someone used the emergency token to disable security. Review audit logs."

      - alert: EmergencyTokenBruteForce
        expr: increase(charon_emergency_token_attempts_total{result="failure"}[5m]) > 10
        labels:
          severity: warning
        annotations:
          summary: "Multiple failed emergency token attempts detected"
          description: "Possible brute force attack on emergency endpoint."

Validation & Testing

Configuration Validation

# Validate docker-compose.yml syntax
docker-compose config

# Verify environment variables are set
docker-compose config | grep EMERGENCY

# Test container starts successfully
docker-compose up -d
docker logs charon | grep -i emergency

Functional Testing

Test Tier 1:

# Test emergency token works
curl -X POST https://charon.example.com/api/v1/emergency/security-reset \
  -H "X-Emergency-Token: $CHARON_EMERGENCY_TOKEN"

# Expected: {"success":true, ...}

Test Tier 2:

# Create SSH tunnel
ssh -L 2020:localhost:2020 admin@server &

# Test emergency server health
curl http://localhost:2020/health

# Test emergency endpoint
curl -X POST http://localhost:2020/emergency/security-reset \
  -H "X-Emergency-Token: $CHARON_EMERGENCY_TOKEN" \
  -u admin:password

# Close tunnel
kill %1

  • Emergency Lockout Recovery Runbook (internal document)
  • Emergency Token Rotation (internal document)
  • Security Documentation

Version History:

  • v1.0 (2026-01-26): Initial release