Skip to content

E2E Test Troubleshooting

Common issues and solutions for Playwright E2E tests.


Recent Improvements (2026-02)

Test Timeout Issues - RESOLVED

Symptoms: Tests timing out after 30 seconds, config reload overlay blocking interactions

Resolution:

  • Extended timeout from 30s to 60s for feature flag propagation
  • Added automatic detection and waiting for config reload overlay
  • Improved test isolation with proper cleanup in afterEach hooks

If you still experience timeouts:

  1. Rebuild the E2E container: .github/skills/scripts/skill-runner.sh docker-rebuild-e2e
  2. Check Docker logs for health check failures
  3. Verify emergency token is set in .env file

API Key Format Mismatch - RESOLVED

Symptoms: Feature flag tests failing with propagation timeout

Resolution:

  • Added key normalization to handle both feature.cerberus.enabled and cerberus.enabled formats
  • Tests now automatically detect and adapt to API response format

Configuration: No manual configuration needed, normalization is automatic.


Quick Diagnostics

Run these commands first:

# Check emergency token is set
grep CHARON_EMERGENCY_TOKEN .env

# Verify token length
echo -n "$(grep CHARON_EMERGENCY_TOKEN .env | cut -d= -f2)" | wc -c
# Should output: 64

# Check Docker container is running
docker ps | grep charon

# Check health endpoint
curl -f http://localhost:8080/api/v1/health || echo "Health check failed"

Error: "CHARON_EMERGENCY_TOKEN is not set"

Symptoms

  • Tests fail immediately with environment configuration error
  • Error appears in global setup before any tests run

Cause

Emergency token not configured in .env file.

Solution

  1. Generate token:
openssl rand -hex 32
  1. Add to .env file:
echo "CHARON_EMERGENCY_TOKEN=<paste_token_here>" >> .env
  1. Verify:
grep CHARON_EMERGENCY_TOKEN .env
  1. Run tests:
npx playwright test --project=chromium

📖 More Info: See Getting Started - Emergency Token Configuration


Error: "CHARON_EMERGENCY_TOKEN is too short"

Symptoms

  • Global setup fails with message about token length
  • Current token length shown in error (e.g., "32 chars, minimum 64")

Cause

Token is shorter than 64 characters (security requirement).

Solution

  1. Regenerate token with correct length:
openssl rand -hex 32  # Generates 64-char hex string
  1. Update .env file:
sed -i "s/CHARON_EMERGENCY_TOKEN=.*/CHARON_EMERGENCY_TOKEN=<new_token>/" .env
  1. Verify length:
echo -n "$(grep CHARON_EMERGENCY_TOKEN .env | cut -d= -f2)" | wc -c
# Should output: 64

Error: "Failed to reset security modules using emergency token"

Symptoms

  • Security teardown fails
  • Causes 20+ cascading test failures
  • Error message about emergency reset

Possible Causes

  1. Token too short (< 64 chars)
  2. Token doesn't match backend configuration
  3. Backend not running or unreachable
  4. Network/container issues

Solution

Step 1: Verify token configuration

# Check token exists and is 64 chars
echo -n "$(grep CHARON_EMERGENCY_TOKEN .env | cut -d= -f2)" | wc -c

# Check backend env matches (if using Docker)
docker exec charon env | grep CHARON_EMERGENCY_TOKEN

Step 2: Verify backend is running

curl http://localhost:8080/api/v1/health
# Should return: {"status":"ok"}

Step 3: Test emergency endpoint directly

curl -X POST http://localhost:8080/api/v1/emergency/security-reset \
  -H "X-Emergency-Token: $(grep CHARON_EMERGENCY_TOKEN .env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d '{"reason":"manual test"}' | jq

Step 4: Check backend logs

# Docker Compose
docker compose logs charon | tail -50

# Docker Run
docker logs charon | tail -50

Step 5: Regenerate token if needed

# Generate new token
NEW_TOKEN=$(openssl rand -hex 32)

# Update .env
sed -i "s/CHARON_EMERGENCY_TOKEN=.*/CHARON_EMERGENCY_TOKEN=${NEW_TOKEN}/" .env

# Restart backend with new token
docker restart charon

# Wait for health
sleep 5 && curl http://localhost:8080/api/v1/health

Error: "Blocked by access control list" (403)

Symptoms

  • Most tests fail with 403 Forbidden errors
  • Error message contains "Blocked by access control"

Cause

Security teardown did not successfully disable ACL before tests ran.

Solution

  1. Run teardown script manually:
npx playwright test tests/security-teardown.setup.ts
  1. Check teardown output for errors:
  2. Look for "Emergency reset successful" message
  3. Verify no error messages about missing token

  4. Verify ACL is disabled:

curl http://localhost:8080/api/v1/security/status | jq
# acl.enabled should be false
  1. If still blocked, manually disable via API:
# Using emergency token
curl -X POST http://localhost:8080/api/v1/emergency/security-reset \
  -H "X-Emergency-Token: $(grep CHARON_EMERGENCY_TOKEN .env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d '{"reason":"manual disable before tests"}'
  1. Run tests again:
npx playwright test --project=chromium

Tests Pass Locally but Fail in CI/CD

Symptoms

  • Tests work on your machine
  • Same tests fail in GitHub Actions
  • Error about missing emergency token in CI logs

Cause

CHARON_EMERGENCY_TOKEN not configured in GitHub Secrets.

Solution

  1. Navigate to repository settings:
  2. Go to: https://github.com/<your-org>/<your-repo>/settings/secrets/actions
  3. Or: Repository → Settings → Secrets and Variables → Actions

  4. Create secret:

  5. Click "New repository secret"
  6. Name: CHARON_EMERGENCY_TOKEN
  7. Value: Generate with openssl rand -hex 32
  8. Click "Add secret"

  9. Verify secret is set:

  10. Secret should appear in list (value is masked)
  11. Cannot view value after creation (security)

  12. Re-run workflow:

  13. Navigate to Actions tab
  14. Re-run failed workflow
  15. Check "Validate Emergency Token Configuration" step passes

📖 Detailed Instructions: See the GitHub Setup Guide (internal document)


Error: "ECONNREFUSED" or "ENOTFOUND"

Symptoms

  • Tests fail with connection refused errors
  • Cannot reach localhost:8080 or configured base URL

Cause

Backend container not running or not accessible.

Solution

  1. Check container status:
docker ps | grep charon
  1. If not running, start it:
# Docker Compose
docker compose up -d

# Docker Run
docker start charon
  1. Wait for health:
timeout 60 bash -c 'until curl -f http://localhost:8080/api/v1/health; do sleep 2; done'
  1. Check logs if still failing:
docker logs charon | tail -50

Error: Token appears to be a placeholder value

Symptoms

  • Global setup validation fails
  • Error mentions "placeholder value"

Cause

Token contains common placeholder strings like:

  • test-emergency-token
  • your_64_character
  • replace_this
  • 0000000000000000

Solution

  1. Generate a unique token:
openssl rand -hex 32
  1. Replace placeholder in .env:
sed -i "s/CHARON_EMERGENCY_TOKEN=.*/CHARON_EMERGENCY_TOKEN=<new_token>/" .env
  1. Verify it's not a placeholder:
grep CHARON_EMERGENCY_TOKEN .env
# Should show a random hex string

Debug Mode

Run tests with full debugging for deeper investigation:

With Playwright Inspector

npx playwright test --debug

Interactive UI for stepping through tests.

With Full Traces

npx playwright test --trace=on

Capture execution traces for each test.

View Trace After Test

npx playwright show-trace test-results/traces/*.zip

Opens trace viewer in browser.

With Enhanced Logging

DEBUG=charon:*,charon-test:* PLAYWRIGHT_DEBUG=1 npx playwright test --project=chromium

Enables all debug output.


Performance Issues

Tests Running Slowly

Symptoms: Tests take > 5 minutes for full suite.

Solutions:

  1. Use sharding (parallel execution):
npx playwright test --shard=1/4 --project=chromium
  1. Run specific test files:
npx playwright test tests/manual-dns-provider.spec.ts
  1. Skip slow tests during development:
npx playwright test --grep-invert "@slow"

Feature Flag Toggle Tests Timing Out

Symptoms:

  • Tests in tests/settings/system-settings.spec.ts fail with timeout errors
  • Error messages mention feature flag toggles (Cerberus, CrowdSec, Uptime, Persist)

Cause:

  • Backend N+1 query pattern causing 300-600ms latency in CI
  • Hard-coded waits insufficient for slower CI environments

Solution (Fixed in v2.x):

  • Backend now uses batch query pattern (3-6x faster: 600ms → 200ms P99)
  • Tests use condition-based polling with waitForFeatureFlagPropagation()
  • Retry logic with exponential backoff handles transient failures

If you still experience issues:

  1. Check backend latency: grep "[METRICS]" docker logs charon
  2. Verify batch query is being used (should see WHERE key IN (...) in logs)
  3. Ensure you're running latest version with the optimization

📖 See Also: Feature Flags Performance Documentation (internal document)

Container Startup Slow

Symptoms: Health check timeouts, tests fail before running.

Solutions:

  1. Increase health check timeout:
timeout 120 bash -c 'until curl -f http://localhost:8080/api/v1/health; do sleep 2; done'
  1. Pre-pull Docker image:
docker pull wikid82/charon:latest
  1. Check Docker resource limits:
docker stats charon
# Ensure adequate CPU/memory

Getting Help

If you're still stuck after trying these solutions:

  1. Check known issues:
  2. Search GitHub Issues

  3. Collect diagnostic info:

# Environment
echo "OS: $(uname -a)"
echo "Docker: $(docker --version)"
echo "Node: $(node --version)"

# Configuration
echo "Base URL: ${PLAYWRIGHT_BASE_URL:-http://localhost:8080}"
echo "Token set: $([ -n "$CHARON_EMERGENCY_TOKEN" ] && echo "Yes" || echo "No")"

# Logs
docker logs charon > charon-logs.txt
npx playwright test --project=chromium > test-output.txt 2>&1
  1. Open GitHub issue:
  2. Include diagnostic info above
  3. Attach charon-logs.txt and test-output.txt
  4. Describe steps to reproduce
  5. Tag with testing and e2e labels

  6. Ask in community:

  7. GitHub Discussions
  8. Include relevant error messages (mask any secrets!)


Last Updated: 2026-02-02