Skip to content

Incident Response

Purpose

What to do when something in production is actually broken, given this repository's specific tooling (no automated monitoring/alerting, no staging tier, no automated rollback).

Architecture/context

See Monitoring for what detection tooling exists (mainly: the deploy-time smoke test, and manual wrangler tail/dashboard checks) - there is no alerting, so detection is frequently a human noticing.

Step-by-step

  1. Confirm it's real. 
    npm run smoke:production                                            # [READ-ONLY]
curl -sI https://<affected-domain>                                  # [READ-ONLY]
  2. Scope it. One app, or everything? One app pointing at Verification's table narrows it to a single Worker; everything failing suggests a Cloudflare-account-wide or DNS issue rather than this repository's code.
  3. Check recent changes. 
    git log --oneline -10 main                                          # [READ-ONLY]
    Compare against the Actions tab's deploy-production history to find the most recent deploy.
  4. Read live logs for the affected Worker - see Logs.
  5. Decide: rollback now, or fix-forward?
  6. If the cause is a bad deploy and a good prior commit is known: rollback first (see Rollback), investigate after. Don't debug in production while it's down.
  7. If the cause is external (Cloudflare outage, DNS, expired token) rollback won't help - see the troubleshooting table below and Cloudflare → Troubleshooting.
  8. After mitigation, write down what happened - this repository has no incident-tracking tool configured; a short note in the PR/commit that fixes it is the available record.

Common causes and first checks

Symptom First check
One app down, others fine That app's wrangler.jsonc routes/build, and its own deploy log
API down, apps still load (stale data) API Worker deploy/D1 issue - see D1 Operations
Everything down at once Cloudflare status page, not this repository's code
Deploy workflow failing in CI CI/CD → Deployment troubleshooting, check which step failed
Works for some users, not others Edge cache staleness - hard refresh before assuming a code issue
Suspected leaked credential Infrastructure → Security - rotate immediately, see CI/CD → Secrets

Rollback

See Rollback for the exact commands.

Security considerations

If the incident involves a leaked secret, rotating it takes priority over root-causing the outage - see CI/CD → Secrets "Security considerations": a git revert does not invalidate a value already exposed in history/logs.