Troubleshooting WebSocket Issues
Troubleshooting WebSocket Issues¶
WebSocket connections are used in Charon for real-time features like live log streaming. If you're experiencing issues with WebSocket connections (e.g., logs not updating in real-time), this guide will help you diagnose and resolve the problem.
Quick Diagnostics¶
Check WebSocket Connection Status¶
- Go to System Settings in the Charon UI
- Scroll to the WebSocket Connections card
- Check if there are active connections displayed
The WebSocket status card shows:
- Total number of active WebSocket connections
- Breakdown by type (General Logs vs Security Logs)
- Oldest connection age
- Detailed connection info (when expanded)
Browser Console Check¶
Open your browser's Developer Tools (F12) and check the Console tab for:
- WebSocket connection errors
- Connection refused messages
- Authentication failures
- CORS errors
Common Issues and Solutions¶
1. Proxy/Load Balancer Configuration¶
Symptom: WebSocket connections fail to establish or disconnect immediately.
Cause: If running Charon behind a reverse proxy (Nginx, Apache, HAProxy, or load balancer), the proxy might be terminating WebSocket connections or not forwarding the upgrade request properly.
Solution:
Nginx Configuration¶
location /api/v1/logs/live {
proxy_pass http://charon:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Increase timeouts for long-lived connections
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
location /api/v1/cerberus/logs/ws {
proxy_pass http://charon:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Increase timeouts for long-lived connections
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
Key requirements:
proxy_http_version 1.1— Required for WebSocket supportUpgradeandConnectionheaders — Required for WebSocket upgrade- Long
proxy_read_timeout— Prevents connection from timing out
Apache Configuration¶
<VirtualHost *:443>
ServerName charon.example.com
# Enable WebSocket proxy
ProxyRequests Off
ProxyPreserveHost On
# WebSocket endpoints
ProxyPass /api/v1/logs/live ws://localhost:8080/api/v1/logs/live retry=0 timeout=3600
ProxyPassReverse /api/v1/logs/live ws://localhost:8080/api/v1/logs/live
ProxyPass /api/v1/cerberus/logs/ws ws://localhost:8080/api/v1/cerberus/logs/ws retry=0 timeout=3600
ProxyPassReverse /api/v1/cerberus/logs/ws ws://localhost:8080/api/v1/cerberus/logs/ws
# Regular HTTP endpoints
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
</VirtualHost>
Required modules:
2. Network Timeouts¶
Symptom: WebSocket connections work initially but disconnect after some idle time.
Cause: Intermediate network infrastructure (firewalls, load balancers, NAT devices) may have idle timeout settings shorter than the WebSocket keepalive interval.
Solution:
Charon sends WebSocket ping frames every 30 seconds to keep connections alive. If you're still experiencing timeouts:
- Check proxy timeout settings (see above)
- Check firewall idle timeout:
# Linux iptables
iptables -L -v -n | grep ESTABLISHED
# If timeout is too short, increase it:
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
echo 3600 > /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established
- Check load balancer settings:
- AWS ALB/ELB: Set idle timeout to 3600 seconds
- GCP Load Balancer: Set timeout to 1 hour
- Azure Load Balancer: Set idle timeout to maximum
3. HTTPS Certificate Errors (Docker)¶
Symptom: WebSocket connections fail with TLS/certificate errors, especially in Docker environments.
Cause: Missing CA certificates in the Docker container, or self-signed certificates not trusted by the browser.
Solution:
Install CA Certificates (Docker)¶
Add to your Dockerfile:
Or for existing containers:
For Self-Signed Certificates (Development Only)¶
Warning: This compromises security. Only use in development environments.
Set environment variable:
Or in docker-compose.yml:
Better Solution: Use Valid Certificates¶
- Use Let's Encrypt (free, automated)
- Use a trusted CA certificate
- Import your self-signed cert into the browser's trust store
4. Firewall Settings¶
Symptom: WebSocket connections fail or time out.
Cause: Firewall blocking WebSocket traffic on ports 80/443.
Solution:
Linux (iptables)¶
Allow WebSocket traffic:
# Allow HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
# Allow established connections (for WebSocket)
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Save rules
iptables-save > /etc/iptables/rules.v4
Docker¶
Ensure ports are exposed:
Cloud Providers¶
- AWS: Add inbound rules to Security Group for ports 80/443
- GCP: Add firewall rules for ports 80/443
- Azure: Add Network Security Group rules for ports 80/443
5. Connection Stability / Packet Loss¶
Symptom: Frequent WebSocket disconnections and reconnections.
Cause: Unstable network with packet loss prevents WebSocket connections from staying open.
Solution:
Check Network Stability¶
# Ping test
ping -c 100 charon.example.com
# Check packet loss (should be < 1%)
mtr charon.example.com
Enable Connection Retry (Client-Side)¶
The Charon frontend automatically handles reconnection for security logs but not general logs. If you need more robust reconnection:
- Monitor the WebSocket status in System Settings
- Refresh the page if connections are frequently dropping
- Consider using a more stable network connection
- Check if VPN or proxy is causing issues
6. Browser Compatibility¶
Symptom: WebSocket connections don't work in certain browsers.
Cause: Very old browsers don't support WebSocket protocol.
Supported Browsers:
- Chrome 16+ ✅
- Firefox 11+ ✅
- Safari 7+ ✅
- Edge (all versions) ✅
- IE 10+ ⚠️ (deprecated, use Edge)
Solution: Update to a modern browser.
7. CORS Issues¶
Symptom: Browser console shows CORS errors with WebSocket connections.
Cause: Cross-origin WebSocket connection blocked by browser security policy.
Solution:
WebSocket connections should be same-origin (from the same domain as the Charon UI). If you're accessing Charon from a different domain:
- Preferred: Access Charon UI from the same domain
- Alternative: Configure CORS in Charon (if supported)
- Development Only: Use browser extension to disable CORS (NOT for production)
8. Authentication Issues¶
Symptom: WebSocket connection fails with 401 Unauthorized.
Cause: Authentication token not being sent with WebSocket connection.
Solution:
Charon WebSocket endpoints support three authentication methods:
- HttpOnly Cookie (automatic) — Used by default when accessing UI from browser
- Query Parameter —
?token=<your-token> - Authorization Header — Not supported for browser WebSocket connections
If you're accessing WebSocket from a script or tool:
9. Live Views Refuse to Connect Behind a Proxy (403)¶
Symptom: The live logs or live stats view never starts, or shows a 403 error. Your other pages work.
Cause: Your proxy rewrites the web address (the Host header), for example nginx with proxy_set_header Host $proxy_host. Charon sees an address that doesn't match the page you are on, and refuses the connection. This happens when no Application URL is set and your proxy is not listed in CHARON_TRUSTED_PROXIES.
Solution: Use one of these:
- Set the Application URL to the address you type in your browser. See Application URL.
- Or add your proxy's address to
CHARON_TRUSTED_PROXIES, then restart Charon. See Trusted Proxies.
10. "Too Many Live Connections" Message¶
Symptom: A live view stops and shows "Too many live connections are open for your account." It does not reconnect by itself.
Cause: Charon allows 8 live views per user, and 64 across all users. Each open live log, Cerberus log, Hecate log or stats view uses one slot. A new view is refused once your slots are full.
Solution: Close another live view (in another tab or window), then open this one again. Other users are not affected.
Monitoring WebSocket Connections¶
Using the System Settings UI¶
- Navigate to System Settings in Charon
- View the WebSocket Connections card
- Expand details to see:
- Connection ID
- Connection type (General/Security)
- Remote address
- Active filters
- Connection duration
Using the API¶
Check WebSocket statistics programmatically:
# Get connection statistics
curl -H "Authorization: Bearer YOUR_TOKEN" \
https://charon.example.com/api/v1/websocket/stats
# Get detailed connection list
curl -H "Authorization: Bearer YOUR_TOKEN" \
https://charon.example.com/api/v1/websocket/connections
Response example:
{
"total_active": 2,
"logs_connections": 1,
"cerberus_connections": 1,
"oldest_connection": "2024-01-15T10:30:00Z",
"last_updated": "2024-01-15T11:00:00Z"
}
Using Browser DevTools¶
- Open DevTools (F12)
- Go to Network tab
- Filter by WS (WebSocket)
- Look for connections to:
/api/v1/logs/live/api/v1/cerberus/logs/ws
Check:
- Status should be
101 Switching Protocols - Messages tab shows incoming log entries
- No errors in Frames tab
Still Having Issues?¶
If none of the above solutions work:
- Check Charon logs:
-
Enable debug logging (if available)
-
Report an issue on GitHub:
- Charon Issues
- Include:
- Charon version
- Browser and version
- Proxy/load balancer configuration
- Error messages from browser console
- Charon server logs