Incident-Response & Alert-Routing

The severity matrix in docs/engineering/operations/incident-response.md formalises Sev1/Sev2/Sev3 with explicit escalation paths, response times and SLA wording so an on-call engineer at 3 a.m. does not have to invent triage. Sev1 is total or partial outage with revenue or data integrity impact and a 15-minute response target; Sev2 is degraded service with workaround; Sev3 is non-urgent. The on-call rotation in docs/engineering/operations/on-call-rotation.md publishes the schedule, contact details and vacation handling so paging never lands on an unreachable number. Every Sev1 and Sev2 closes with a blameless postmortem from the docs/engineering/operations/postmortem-template.md, including timeline, root cause, contributing factors and an action list with owners.

Alerting runs through Azure Monitor. Action group ag-petanque-primary (email + Slack webhook) is provisioned via infrastructure/monitoring.bicep and applied with infrastructure/deploy-monitoring.sh, so the channel topology is in version control. Sev1 alert rules cover 5xx-rate > 1 % over 5 minutes, Cosmos RU throttling > 10 % over 5 minutes and ACS email bounce rate > 5 % over 15 minutes — failures that demand immediate attention. Sev2 covers Container App restarts > 3/h, API latency p95 > 2 s over 10 minutes and Stripe webhook errors > 5 % — degraded but not bleeding.

The incident lifecycle is captured server-side in IncidentLog (a Beanie model) with severity, workflow status (investigating / identified / monitoring / resolved), public_message and an updates list. Platform admins create and update incidents through POST /admin/incidents and PATCH /admin/incidents/{id}; updates are append-only on the embedded list so the public timeline is never edited after the fact. GET /public/incidents?since=<iso> serves the last 30 days of public incidents, sorted newest-first, with only public fields exposed. The status page (www/src/pages/status.astro plus fr/es/sv variants) reads this endpoint and renders severity and workflow badges so users see the same picture the on-call sees, without authentication.