Uptime Monitoring¶
Charon's uptime monitoring system continuously checks the availability of your proxy hosts and alerts you when issues occur. The system is designed to minimize false positives while quickly detecting real problems.
Overview¶
Uptime monitoring performs automated health checks on your proxy hosts at regular intervals, tracking:
- Host availability (TCP connectivity)
- Response times (latency measurements)
- Status history (uptime/downtime tracking)
- Failure patterns (debounced detection)
How It Works¶
Check Cycle¶
- Scheduled Checks: Each monitor runs on its own interval (60 seconds by default, minimum 30 seconds)
- Port Detection: Uses the proxy host's
ForwardPortfor TCP checks - Connection Test: Attempts TCP connection with configurable timeout
- Status Update: Records success/failure in database
- Notification Trigger: Sends alerts on status changes (if configured)
Failure Debouncing¶
To prevent false alarms from transient network issues, Charon uses failure debouncing:
How it works:
- A host must fail 2 consecutive checks before being marked "down"
- Single failures are logged but don't trigger status changes
- Counter resets immediately on any successful check
Why this matters:
- Network hiccups don't cause false alarms
- Container restarts don't trigger unnecessary alerts
- Transient DNS issues are ignored
- You only get notified about real problems
Example scenario:
Check 1: ✅ Success → Status: Up, Failure Count: 0
Check 2: ❌ Failed → Status: Up, Failure Count: 1 (no alert)
Check 3: ❌ Failed → Status: Down, Failure Count: 2 (alert sent!)
Check 4: ✅ Success → Status: Up, Failure Count: 0 (recovery alert)
Configuration¶
Timeout Settings¶
Connection timeout: 3 seconds
This is how long Charon waits for a connection to a monitor's address before it counts that check as failed.
Increase timeout if:
- You have slow networks
- Hosts are geographically distant
- Containers take time to warm up
- You see intermittent false "down" alerts
Decrease timeout if:
- You want faster failure detection
- Your hosts are on local network
- Response times are consistently fast
Note: Timeout settings are currently set in the backend configuration. A future release will make this configurable via the UI.
Retry Behavior¶
A single check is one attempt. Rather than retrying in a tight loop, Charon waits for the next scheduled check and only changes a monitor's status once the result has been consistent:
- Consecutive failures before "down": 2 (a monitor must fail 2 checks in a row before it is reported down; one recovery check flips it back to "up")
- Timeout per check: capped at the monitor's interval, up to 20 seconds
- Connection timeout: 3 seconds
Because failures are counted across scheduled checks instead of retried back-to-back, a monitor on a 30-second interval is reported down roughly 60 seconds after it actually goes down.
Check Interval (Per Monitor)¶
Default: 60 seconds Minimum: 30 seconds
Every monitor has its own check interval. You set it in the monitor's Add or Edit form, so a critical service can be checked every 30 seconds while a less important one is checked every few minutes.
What to expect:
- Monitors created before this update keep checking every 60 seconds until you open them, change the interval, and save.
- Shorter intervals catch problems sooner but do a little more work.
- Longer intervals are lighter but tell you about an outage later.
- Anything below 30 seconds is rounded up to 30.
Uptime Monitoring Engine Settings¶
Three settings under System Settings → Uptime Monitoring tune the whole engine. Most people never need to touch them.
| Setting | What it does | Range | When it applies |
|---|---|---|---|
| Default check interval | The interval a brand-new monitor starts with | 30 – 86400 seconds | Within about a minute, no restart |
| Worker pool size | How many checks Charon runs at the same time | 1 – 200 | After you restart Charon |
| Heartbeat retention | Days of per-check history to keep before automatic cleanup | 1 – 3650 days (default 30) | Within about an hour, no restart |
If you monitor many services and some of them are often slow or down, raising the Worker pool size (to around 60–90) gives Charon more room to keep every check on schedule.
History Retention¶
Charon records one "heartbeat" for every check it runs, and uses that history to draw your uptime graphs and percentages. To keep the database tidy, a cleanup job runs once an hour and permanently deletes heartbeats older than your Heartbeat retention setting (30 days by default). Pick a window that matches how far back you like to look.
Reclaiming disk space¶
Every check adds a small record, so history takes up room. As a rough guide, expect about 15 MB per monitor per 30 days at 60-second checks.
- Records older than Heartbeat retention are deleted automatically every hour.
- If you lower the setting, the freed space is returned to your disk automatically: Charon trims the database file in the background, and on an older database with a lot of free space it optimizes the file once at the next start. See Database Maintenance for what to expect, how to turn it off, and a manual fallback.
- Upgrading: the first time Charon starts after this change, installs whose retention was still the untouched old default (90 days) are moved once to 30 days, and older history is then trimmed hourly. If you want longer history, raise the setting in System Settings.
Who Can Change Monitors¶
- Administrators can add, edit, delete, pause, and run a manual check on any monitor, and can use Sync with Hosts to create monitors for existing hosts.
- Standard users can view the monitor list, each monitor's status, and its history. They cannot add, change, delete, or check monitors. If a standard user asks for one of these changes, Charon refuses it.
Monitors are checked by Charon itself, so only administrators can point them at new targets.
Which Addresses a Monitor Can Check¶
Monitors can check addresses on your own network:
- This machine (
localhostor127.0.0.1), on ports Charon does not use itself - Home and office networks (
10.x.x.x,172.16.x.xto172.31.x.x,192.168.x.x) - Tailscale and other overlay networks (
100.64.0.0/10) - IPv6 private addresses (
fc00::/7) - Fake-IP addresses used by some VPN and proxy tools (
198.18.0.0/15) - The names Docker Desktop, OrbStack and Podman use for the host machine:
host.docker.internal,host.containers.internalandgateway.docker.internal
Charon will not check:
- Link-local addresses (
169.254.0.0/16andfe80::/10) - Cloud instance service addresses, such as
169.254.169.254,100.100.100.200,168.63.129.16, and the IPv6 addressesfd00:ec2::/32andfd20:ce::254. These are refused even when their wider network is allowed. - Reserved addresses such as
0.0.0.0, multicast and broadcast addresses - The ports Charon uses for itself on this machine: its agent ports, its internal admin port (2019), its bundled CrowdSec port (8085), and the emergency server port while that server is turned on
Monitor types:
- TCP monitors take
host:port, such as192.168.1.20:5432, ortcp://host:port. - HTTP and HTTPS monitors take a full web address, such as
https://nas.local:8443. Without a port, the web default (80 or 443) is used.
When Charon checks an address:
- When you save a monitor, Charon refuses the save with the message "This destination is not allowed." when the address is one it does not check. A name is looked up at that moment, and the monitor is refused if any address it points to is refused. If the lookup fails or takes too long, the monitor is saved anyway.
- Charon checks the address again each time a check connects. If a name later points to a refused address, the check reports the monitor as down.
Monitors saved before a rule applied:
- The monitor is kept, but each check reports it as down with the message "Destination is not allowed for checks". At startup, Charon also writes a warning to its log for each such monitor that it can spot from the address alone, without a name lookup.
- Delete the monitor, or create a new one with another address.
Enabling Uptime Monitoring¶
Turning monitoring on is an administrator task. The steps below are for an administrator account.
For a Single Host¶
- Navigate to Proxy Hosts
- Click Edit on the host
- Scroll to Uptime Monitoring section
- Toggle "Enable Uptime Monitoring" to ON
- Click Save
For Multiple Hosts (Bulk)¶
- Navigate to Proxy Hosts
- Select checkboxes for hosts to monitor
- Click "Bulk Apply" button
- Find "Uptime Monitoring" section
- Toggle the switch to ON
- Check "Apply to selected hosts"
- Click "Apply Changes"
Monitoring Dashboard¶
Host Status Display¶
Each monitored host shows:
- Status Badge: 🟢 Up / 🔴 Down
- Response Time: Last successful check latency
- Uptime Percentage: Success rate over time
- Last Check: Timestamp of most recent check
Status Page¶
View all monitored hosts at a glance:
- Navigate to Dashboard → Uptime Status
- See real-time status of all hosts
- Click any host for detailed history
- Filter by status (up/down/all)
Performance & Scale¶
Running Hundreds of Monitors¶
The uptime engine is built to run hundreds of monitors on a single Charon instance without the dashboard slowing down. The Uptime page loads the current state of every monitor in one request, so adding more monitors does not pile on more page-load work.
The First Start After This Update¶
The first time this version starts on a Charon instance that already has a large check history, Charon does two one-time housekeeping jobs in the background: it clears out history that is past your retention window, and it builds a database index that keeps the Uptime page fast. On instances with a very large history this can take a few minutes, and during that time the Uptime page may respond a little more slowly than usual. It returns to normal on its own once the housekeeping finishes. There is nothing you need to do.
Troubleshooting¶
Monitor Shows "Destination is not allowed for checks"¶
Cause: The monitor's address is one Charon does not check. It was saved before the rule applied, or it points to a link-local, cloud instance service, or Charon-reserved address.
Solution:
- Compare the monitor's address with the list in Which Addresses a Monitor Can Check
- Delete the monitor and create a new one with an address Charon allows, such as the device's address on your local network
False Positive: Host Shown as Down but Actually Up¶
Symptoms:
- Host shows "down" in Charon
- Service is accessible directly
- Status changes back to "up" shortly after
Common causes:
- Timeout too short for slow network
Solution: Increase TCP timeout in configuration
- Container warmup time exceeds timeout
Solution: Use longer timeout or optimize container startup
- Network congestion during check
Solution: Debouncing (already enabled) should handle this automatically
- Firewall blocking health checks
Solution: Ensure Charon container can reach proxy host ports
- Multiple checks running concurrently
Solution: Automatic synchronization ensures checks complete before next cycle
Diagnostic steps:
# Check Charon logs for timing info
docker logs charon 2>&1 | grep "Host TCP check completed"
# Look for retry attempts
docker logs charon 2>&1 | grep "Retrying TCP check"
# Check failure count patterns
docker logs charon 2>&1 | grep "failure_count"
# View host status changes
docker logs charon 2>&1 | grep "Host status changed"
False Negative: Host Shown as Up but Actually Down¶
Symptoms:
- Host shows "up" in Charon
- Service returns errors or is inaccessible
- No down alerts received
Common causes:
- TCP port open but service not responding
Explanation: Uptime monitoring only checks TCP connectivity, not application health
Solution: Consider implementing application-level health checks (future feature)
- Service accepts connections but returns errors
Solution: Monitor application logs separately; TCP checks don't validate responses
- Partial service degradation
Solution: Use multiple monitoring providers for critical services
Current limitation: A TCP monitor only confirms that the port accepts a connection. HTTP and HTTPS monitors also check the status the site answers with: a success or redirect status (200 to 399), or 401 or 403, counts as up. Charon does not read the content of the page.
Intermittent Status Flapping¶
Symptoms:
- Status rapidly changes between up/down
- Multiple notifications in short time
- Logs show alternating success/failure
Causes:
- Marginal network conditions
Solution: Increase failure threshold (requires configuration change)
- Resource exhaustion on target host
Solution: Investigate target host performance, increase resources
- Shared network congestion
Solution: Consider dedicated monitoring network or VLAN
Mitigation:
The built-in debouncing (2 consecutive failures required) should prevent most flapping. If issues persist, check:
# Review consecutive check results
docker logs charon 2>&1 | grep -A 2 "Host TCP check completed" | grep "host_name"
# Check response time trends
docker logs charon 2>&1 | grep "elapsed_ms"
No Notifications Received¶
Checklist:
- ✅ Uptime monitoring is enabled for the host
- ✅ Notification provider is configured and enabled
- ✅ Provider is set to trigger on uptime events
- ✅ Status has actually changed (check logs)
- ✅ Debouncing threshold has been met (2 consecutive failures)
Debug notifications:
# Check for notification attempts
docker logs charon 2>&1 | grep "notification"
# Look for uptime-related notifications
docker logs charon 2>&1 | grep "uptime_down\|uptime_up"
# Verify notification service is working
docker logs charon 2>&1 | grep "Failed to send notification"
High CPU Usage from Monitoring¶
Symptoms:
- Charon container using excessive CPU
- System becomes slow during check cycles
- Logs show slow check times
Solutions:
- Reduce number of monitored hosts
Monitor only critical services; disable monitoring for non-essential hosts
- Increase check interval
Change from 60s to 120s to reduce frequency
- Optimize Docker resource allocation
Ensure adequate CPU/memory allocated to Charon container
- Check for network issues
Slow DNS or network problems can cause checks to hang
- Check the engine health counters
Open GET /api/v1/uptime/health (see API Integration below). If queue_depth
stays high or the dropped counters keep rising, raise the Worker pool size
setting and restart Charon.
Monitor check performance:
# View check duration distribution
docker logs charon 2>&1 | grep "elapsed_ms" | tail -50
# Count concurrent checks
docker logs charon 2>&1 | grep "All host checks completed"
Advanced Topics¶
Port Detection¶
Charon automatically determines which port to check:
Priority order:
- ProxyHost.ForwardPort: Preferred, most reliable
- URL extraction: Fallback for hosts without proxy configuration
- Default ports: 80 (HTTP) or 443 (HTTPS) if port not specified
Example:
Host: example.com
Forward Port: 8080
→ Checks: example.com:8080
Host: api.example.com
URL: https://api.example.com/health
Forward Port: (not set)
→ Checks: api.example.com:443
Concurrent Check Processing¶
All host checks run concurrently for better performance:
- Each host checked in separate goroutine
- WaitGroup ensures all checks complete before next cycle
- Prevents database race conditions
- No single slow host blocks other checks
Performance characteristics:
- Sequential checks (old):
time = hosts × timeout - Concurrent checks (current):
time = max(individual_check_times)
Example: With 10 hosts and 10s timeout:
- Sequential: ~100 seconds minimum
- Concurrent: ~10 seconds (if all succeed on first try)
Database Storage¶
Uptime data is stored efficiently:
UptimeHost table:
status: Current status ("up"/"down")failure_count: Consecutive failure counterlast_check: Timestamp of last checkresponse_time: Last successful response time
UptimeMonitor table:
- Links monitors to proxy hosts
- Stores check configuration
- Tracks enabled state
Heartbeat records:
- One record per check, kept for the number of days set by Heartbeat retention (default 30)
- Used for uptime percentage calculations and history graphs
- Cleaned up automatically once an hour; records older than the retention window are permanently deleted
Best Practices¶
1. Monitor Critical Services Only¶
Don't monitor every host. Focus on:
- Production services
- User-facing applications
- External dependencies
- High-availability requirements
Skip monitoring for:
- Development/test instances
- Internal tools with built-in redundancy
- Services with their own monitoring
2. Configure Appropriate Notifications¶
Critical services:
- Multiple notification channels (Discord + Slack)
- Immediate alerts (no batching)
- On-call team notifications
Non-critical services:
- Single notification channel
- Digest/batch notifications (future feature)
- Email to team (low priority)
3. Review False Positives¶
If you receive false alarms:
- Check logs to understand why
- Adjust timeout if needed
- Verify network stability
- Consider increasing failure threshold (future config option)
4. Regular Status Review¶
Weekly review of:
- Uptime percentages (identify problematic hosts)
- Response time trends (detect degradation)
- Notification frequency (too many alerts?)
- False positive rate (refine configuration)
5. Combine with Application Monitoring¶
Uptime monitoring checks availability, not functionality.
Complement with:
- Application-level health checks
- Error rate monitoring
- Performance metrics (APM tools)
- User experience monitoring
Planned Improvements¶
Future enhancements under consideration:
- [ ] HTTP health check support - Check specific endpoints with status code validation
- [ ] Configurable failure threshold - Adjust consecutive failure count via UI
- [ ] Response time alerts - Notify on degraded performance, not just failures
- [ ] Notification batching - Group multiple alerts to reduce noise
- [ ] Maintenance windows - Disable alerts during scheduled maintenance
- [ ] Historical graphs - Visual uptime trends over time
- [ ] Status page export - Public status page for external visibility
Monitoring the Monitors¶
How do you know if Charon's monitoring is working?
Check Charon's own health:
# Verify check cycle is running
docker logs charon 2>&1 | grep "All host checks completed" | tail -5
# Confirm recent checks happened
docker logs charon 2>&1 | grep "Host TCP check completed" | tail -20
# Look for any errors in monitoring system
docker logs charon 2>&1 | grep "ERROR.*uptime\|ERROR.*monitor"
Expected log pattern:
INFO[...] All host checks completed host_count=5
DEBUG[...] Host TCP check completed elapsed_ms=156 host_name=example.com success=true
Warning signs:
- No "All host checks completed" messages in recent logs
- Checks taking longer than expected (>30s with 10s timeout)
- Frequent timeout errors
- High failure_count values
API Integration¶
Uptime monitoring data is accessible via API:
Get uptime status:
Response:
{
"hosts": [
{
"id": "123",
"name": "example.com",
"status": "up",
"last_check": "2025-12-24T10:30:00Z",
"response_time": 156,
"failure_count": 0,
"uptime_percentage": 99.8
}
]
}
Check the engine's own health:
This returns simple counters for the monitoring engine itself: how many checks are waiting in the queue, the current worker-pool size, and whether any check results or history writes have been dropped because the system was briefly overloaded. All zeros is the healthy state. If the dropped counters climb, raise the Worker pool size setting and restart Charon.
Programmatic monitoring:
Use this API to integrate Charon's uptime data with:
- External monitoring dashboards (Grafana, etc.)
- Incident response systems (PagerDuty, etc.)
- Custom alerting tools
- Status page generators
Additional Resources¶
Monitoring a service on a remote Docker host? See Connecting a Remote Docker Host.