Skip to main content

Uptime Monitoring

Charon's uptime monitoring system continuously checks the availability of your proxy hosts and alerts you when issues occur. The system is designed to minimize false positives while quickly detecting real problems.

Overview​

Uptime monitoring performs automated health checks on your proxy hosts at regular intervals, tracking:

  • Host availability (TCP connectivity)
  • Response times (latency measurements)
  • Status history (uptime/downtime tracking)
  • Failure patterns (debounced detection)

How It Works​

Check Cycle​

  1. Scheduled Checks: Each monitor runs on its own interval (60 seconds by default, minimum 30 seconds)
  2. Port Detection: Uses the proxy host's ForwardPort for TCP checks
  3. Connection Test: Attempts TCP connection with configurable timeout
  4. Status Update: Records success/failure in database
  5. Notification Trigger: Sends alerts on status changes (if configured)

Failure Debouncing​

To prevent false alarms from transient network issues, Charon uses failure debouncing:

How it works:

  • A host must fail 2 consecutive checks before being marked "down"
  • Single failures are logged but don't trigger status changes
  • Counter resets immediately on any successful check

Why this matters:

  • Network hiccups don't cause false alarms
  • Container restarts don't trigger unnecessary alerts
  • Transient DNS issues are ignored
  • You only get notified about real problems

Example scenario:

Check 1: ✅ Success → Status: Up, Failure Count: 0
Check 2: ❌ Failed  → Status: Up, Failure Count: 1  (no alert)
Check 3: ❌ Failed  → Status: Down, Failure Count: 2 (alert sent!)
Check 4: ✅ Success → Status: Up, Failure Count: 0  (recovery alert)

Configuration​

Timeout Settings​

Default TCP timeout: 10 seconds

This timeout determines how long Charon waits for a TCP connection before considering it failed.

Increase timeout if:

  • You have slow networks
  • Hosts are geographically distant
  • Containers take time to warm up
  • You see intermittent false "down" alerts

Decrease timeout if:

  • You want faster failure detection
  • Your hosts are on local network
  • Response times are consistently fast

Note: Timeout settings are currently set in the backend configuration. A future release will make this configurable via the UI.

Retry Behavior​

A single check is one attempt. Rather than retrying in a tight loop, Charon waits for the next scheduled check and only changes a monitor's status once the result has been consistent:

  • Consecutive failures before "down": 2 (a monitor must fail 2 checks in a row before it is reported down; one recovery check flips it back to "up")
  • Timeout per check: capped at the monitor's interval, up to 20 seconds
  • Connection timeout: 3 seconds

Because failures are counted across scheduled checks instead of retried back-to-back, a monitor on a 30-second interval is reported down roughly 60 seconds after it actually goes down.

Check Interval (Per Monitor)​

Default: 60 seconds Minimum: 30 seconds

Every monitor has its own check interval. You set it in the monitor's Add or Edit form, so a critical service can be checked every 30 seconds while a less important one is checked every few minutes.

What to expect:

  • Monitors created before this update keep checking every 60 seconds until you open them, change the interval, and save.
  • Shorter intervals catch problems sooner but do a little more work.
  • Longer intervals are lighter but tell you about an outage later.
  • Anything below 30 seconds is rounded up to 30.

Uptime Monitoring Engine Settings​

Three settings under System Settings → Uptime Monitoring tune the whole engine. Most people never need to touch them.

SettingWhat it doesRangeWhen it applies
Default check intervalThe interval a brand-new monitor starts with30 – 86400 secondsWithin about a minute, no restart
Worker pool sizeHow many checks Charon runs at the same time1 – 200After you restart Charon
Heartbeat retentionDays of per-check history to keep before automatic cleanup1 – 3650 days (default 90)Within about an hour, no restart

If you monitor many services and some of them are often slow or down, raising the Worker pool size (to around 60–90) gives Charon more room to keep every check on schedule.

History Retention​

Charon records one "heartbeat" for every check it runs, and uses that history to draw your uptime graphs and percentages. To keep the database tidy, a cleanup job runs once an hour and permanently deletes heartbeats older than your Heartbeat retention setting (90 days by default). Pick a window that matches how far back you like to look.

Enabling Uptime Monitoring​

For a Single Host​

  1. Navigate to Proxy Hosts
  2. Click Edit on the host
  3. Scroll to Uptime Monitoring section
  4. Toggle "Enable Uptime Monitoring" to ON
  5. Click Save

For Multiple Hosts (Bulk)​

  1. Navigate to Proxy Hosts
  2. Select checkboxes for hosts to monitor
  3. Click "Bulk Apply" button
  4. Find "Uptime Monitoring" section
  5. Toggle the switch to ON
  6. Check "Apply to selected hosts"
  7. Click "Apply Changes"

Monitoring Dashboard​

Host Status Display​

Each monitored host shows:

  • Status Badge: 🟢 Up / 🔴 Down
  • Response Time: Last successful check latency
  • Uptime Percentage: Success rate over time
  • Last Check: Timestamp of most recent check

Status Page​

View all monitored hosts at a glance:

  1. Navigate to Dashboard → Uptime Status
  2. See real-time status of all hosts
  3. Click any host for detailed history
  4. Filter by status (up/down/all)

Performance & Scale​

Running Hundreds of Monitors​

The uptime engine is built to run hundreds of monitors on a single Charon instance without the dashboard slowing down. The Uptime page loads the current state of every monitor in one request, so adding more monitors does not pile on more page-load work.

The First Start After This Update​

The first time this version starts on a Charon instance that already has a large check history, Charon does two one-time housekeeping jobs in the background: it clears out history that is past your retention window, and it builds a database index that keeps the Uptime page fast. On instances with a very large history this can take a few minutes, and during that time the Uptime page may respond a little more slowly than usual. It returns to normal on its own once the housekeeping finishes. There is nothing you need to do.

Troubleshooting​

False Positive: Host Shown as Down but Actually Up​

Symptoms:

  • Host shows "down" in Charon
  • Service is accessible directly
  • Status changes back to "up" shortly after

Common causes:

  1. Timeout too short for slow network

    Solution: Increase TCP timeout in configuration

  2. Container warmup time exceeds timeout

    Solution: Use longer timeout or optimize container startup

  3. Network congestion during check

    Solution: Debouncing (already enabled) should handle this automatically

  4. Firewall blocking health checks

    Solution: Ensure Charon container can reach proxy host ports

  5. Multiple checks running concurrently

    Solution: Automatic synchronization ensures checks complete before next cycle

Diagnostic steps:

# Check Charon logs for timing info
docker logs charon 2>&1 | grep "Host TCP check completed"

# Look for retry attempts
docker logs charon 2>&1 | grep "Retrying TCP check"

# Check failure count patterns
docker logs charon 2>&1 | grep "failure_count"

# View host status changes
docker logs charon 2>&1 | grep "Host status changed"

False Negative: Host Shown as Up but Actually Down​

Symptoms:

  • Host shows "up" in Charon
  • Service returns errors or is inaccessible
  • No down alerts received

Common causes:

  1. TCP port open but service not responding

    Explanation: Uptime monitoring only checks TCP connectivity, not application health

    Solution: Consider implementing application-level health checks (future feature)

  2. Service accepts connections but returns errors

    Solution: Monitor application logs separately; TCP checks don't validate responses

  3. Partial service degradation

    Solution: Use multiple monitoring providers for critical services

Current limitation: Charon performs TCP health checks only. HTTP-based health checks are planned for a future release.

Intermittent Status Flapping​

Symptoms:

  • Status rapidly changes between up/down
  • Multiple notifications in short time
  • Logs show alternating success/failure

Causes:

  1. Marginal network conditions

    Solution: Increase failure threshold (requires configuration change)

  2. Resource exhaustion on target host

    Solution: Investigate target host performance, increase resources

  3. Shared network congestion

    Solution: Consider dedicated monitoring network or VLAN

Mitigation:

The built-in debouncing (2 consecutive failures required) should prevent most flapping. If issues persist, check:

# Review consecutive check results
docker logs charon 2>&1 | grep -A 2 "Host TCP check completed" | grep "host_name"

# Check response time trends
docker logs charon 2>&1 | grep "elapsed_ms"

No Notifications Received​

Checklist:

  1. ✅ Uptime monitoring is enabled for the host
  2. ✅ Notification provider is configured and enabled
  3. ✅ Provider is set to trigger on uptime events
  4. ✅ Status has actually changed (check logs)
  5. ✅ Debouncing threshold has been met (2 consecutive failures)

Debug notifications:

# Check for notification attempts
docker logs charon 2>&1 | grep "notification"

# Look for uptime-related notifications
docker logs charon 2>&1 | grep "uptime_down\|uptime_up"

# Verify notification service is working
docker logs charon 2>&1 | grep "Failed to send notification"

High CPU Usage from Monitoring​

Symptoms:

  • Charon container using excessive CPU
  • System becomes slow during check cycles
  • Logs show slow check times

Solutions:

  1. Reduce number of monitored hosts

    Monitor only critical services; disable monitoring for non-essential hosts

  2. Increase check interval

    Change from 60s to 120s to reduce frequency

  3. Optimize Docker resource allocation

    Ensure adequate CPU/memory allocated to Charon container

  4. Check for network issues

    Slow DNS or network problems can cause checks to hang

  5. Check the engine health counters

    Open GET /api/v1/uptime/health (see API Integration below). If queue_depth stays high or the dropped counters keep rising, raise the Worker pool size setting and restart Charon.

Monitor check performance:

# View check duration distribution
docker logs charon 2>&1 | grep "elapsed_ms" | tail -50

# Count concurrent checks
docker logs charon 2>&1 | grep "All host checks completed"

Advanced Topics​

Port Detection​

Charon automatically determines which port to check:

Priority order:

  1. ProxyHost.ForwardPort: Preferred, most reliable
  2. URL extraction: Fallback for hosts without proxy configuration
  3. Default ports: 80 (HTTP) or 443 (HTTPS) if port not specified

Example:

Host: example.com
Forward Port: 8080
→ Checks: example.com:8080

Host: api.example.com
URL: https://api.example.com/health
Forward Port: (not set)
→ Checks: api.example.com:443

Concurrent Check Processing​

All host checks run concurrently for better performance:

  • Each host checked in separate goroutine
  • WaitGroup ensures all checks complete before next cycle
  • Prevents database race conditions
  • No single slow host blocks other checks

Performance characteristics:

  • Sequential checks (old): time = hosts × timeout
  • Concurrent checks (current): time = max(individual_check_times)

Example: With 10 hosts and 10s timeout:

  • Sequential: ~100 seconds minimum
  • Concurrent: ~10 seconds (if all succeed on first try)

Database Storage​

Uptime data is stored efficiently:

UptimeHost table:

  • status: Current status ("up"/"down")
  • failure_count: Consecutive failure counter
  • last_check: Timestamp of last check
  • response_time: Last successful response time

UptimeMonitor table:

  • Links monitors to proxy hosts
  • Stores check configuration
  • Tracks enabled state

Heartbeat records:

  • One record per check, kept for the number of days set by Heartbeat retention (default 90)
  • Used for uptime percentage calculations and history graphs
  • Cleaned up automatically once an hour; records older than the retention window are permanently deleted

Best Practices​

1. Monitor Critical Services Only​

Don't monitor every host. Focus on:

  • Production services
  • User-facing applications
  • External dependencies
  • High-availability requirements

Skip monitoring for:

  • Development/test instances
  • Internal tools with built-in redundancy
  • Services with their own monitoring

2. Configure Appropriate Notifications​

Critical services:

  • Multiple notification channels (Discord + Slack)
  • Immediate alerts (no batching)
  • On-call team notifications

Non-critical services:

  • Single notification channel
  • Digest/batch notifications (future feature)
  • Email to team (low priority)

3. Review False Positives​

If you receive false alarms:

  1. Check logs to understand why
  2. Adjust timeout if needed
  3. Verify network stability
  4. Consider increasing failure threshold (future config option)

4. Regular Status Review​

Weekly review of:

  • Uptime percentages (identify problematic hosts)
  • Response time trends (detect degradation)
  • Notification frequency (too many alerts?)
  • False positive rate (refine configuration)

5. Combine with Application Monitoring​

Uptime monitoring checks availability, not functionality.

Complement with:

  • Application-level health checks
  • Error rate monitoring
  • Performance metrics (APM tools)
  • User experience monitoring

Planned Improvements​

Future enhancements under consideration:

  • HTTP health check support - Check specific endpoints with status code validation
  • Configurable failure threshold - Adjust consecutive failure count via UI
  • Response time alerts - Notify on degraded performance, not just failures
  • Notification batching - Group multiple alerts to reduce noise
  • Maintenance windows - Disable alerts during scheduled maintenance
  • Historical graphs - Visual uptime trends over time
  • Status page export - Public status page for external visibility

Monitoring the Monitors​

How do you know if Charon's monitoring is working?

Check Charon's own health:

# Verify check cycle is running
docker logs charon 2>&1 | grep "All host checks completed" | tail -5

# Confirm recent checks happened
docker logs charon 2>&1 | grep "Host TCP check completed" | tail -20

# Look for any errors in monitoring system
docker logs charon 2>&1 | grep "ERROR.*uptime\|ERROR.*monitor"

Expected log pattern:

INFO[...] All host checks completed host_count=5
DEBUG[...] Host TCP check completed elapsed_ms=156 host_name=example.com success=true

Warning signs:

  • No "All host checks completed" messages in recent logs
  • Checks taking longer than expected (>30s with 10s timeout)
  • Frequent timeout errors
  • High failure_count values

API Integration​

Uptime monitoring data is accessible via API:

Get uptime status:

GET /api/uptime/hosts
Authorization: Bearer <token>

Response:

{
  "hosts": [
    {
      "id": "123",
      "name": "example.com",
      "status": "up",
      "last_check": "2025-12-24T10:30:00Z",
      "response_time": 156,
      "failure_count": 0,
      "uptime_percentage": 99.8
    }
  ]
}

Check the engine's own health:

GET /api/v1/uptime/health
Authorization: Bearer <token>

This returns simple counters for the monitoring engine itself: how many checks are waiting in the queue, the current worker-pool size, and whether any check results or history writes have been dropped because the system was briefly overloaded. All zeros is the healthy state. If the dropped counters climb, raise the Worker pool size setting and restart Charon.

Programmatic monitoring:

Use this API to integrate Charon's uptime data with:

  • External monitoring dashboards (Grafana, etc.)
  • Incident response systems (PagerDuty, etc.)
  • Custom alerting tools
  • Status page generators

Additional Resources​

Monitoring a service on a remote Docker host? See Connecting a Remote Docker Host.

Need Help?​