Skip to main content

Runbook: API Down

Trigger

Use this runbook when backend APIs are unavailable, timing out, returning 5xx errors, or rejecting valid authenticated requests.

Immediate Actions

StepActionOwnerEvidenceStatus
1Identify affected endpoints and clientsBackend / QAEndpoint list and error responseTBD
2Test public health endpoint or known safe endpointOperationsHTTP status evidenceTBD
3Test protected endpoint with valid tokenBackend / QAAPI response evidenceTBD
4Check recent deployment or config changeRelease OwnerDeployment recordTBD
5Check Laravel logs and SentryBackendSanitized evidenceTBD
6Check database, Redis, queue, and upstream service healthOperationsHealth evidenceTBD

Common Causes

CauseCheck
Bad deploymentCompare deployment time with failure start
Database connectivity issueCheck DB health and application errors
Redis or queue failureCheck queue worker and Redis status
Auth or token issueCompare login and protected endpoint behavior
Rate limiting or WAF ruleCheck Cloudflare and application logs

Recovery Criteria

  • Affected endpoints return expected status codes.
  • Valid authenticated users can access protected endpoints.
  • No repeated critical exceptions in Sentry.
  • QA smoke tests pass for login and one protected flow.