Shipflash logoDocs

Operational Readiness And Recovery

Define service signals, incident ownership, application rollback, database forward repair, backups, restore drills, and release evidence before customers depend on the product.

On this page

Minimum production signals

SignalWatchFirst investigation
Liveness and readiness/health.txt separately from /api/health/ready.Deployment status, application logs, Supabase health, and database connectivity.
AuthenticationSign-in/callback failures and unusual denial rates.Supabase Auth configuration, callback origin, session cookies, and active profile lifecycle.
BillingCheckout errors, webhook failures, stale processing claims, projection lag, refund failures, and provider API errors.Provider event ID, request ID, local webhook record, customer, subscription, and entitlement state.
NotificationsOutbox depth, oldest queued age, retries, dead letters, send failures, and Resend webhook failures.Outbox ID, event key, recipient resolution, attempt history, provider message ID, and scheduler execution.
Scheduled workLast successful publish, notification-outbox, and retention run.Scheduler method, bearer secret, response, job logs, rate limit, and claimed work.
Growing dataDatabase latency, rows scanned/returned, export size, memory, connection pressure, and timeouts.Query bounds, indexes, server-side aggregation, pagination, and production-shaped query plan.
  1. 1

    Declare severity and ownership

    Treat security exposure, incorrect billing, widespread auth failure, or unrecoverable data loss as the highest severity. Assign one incident owner and one communication owner.

  2. 2

    Stabilize integrity

    Record start time, confirmed impact, affected systems, and stable identifiers. Stop unsafe writes or provider actions when data or financial integrity is uncertain.

  3. 3

    Diagnose from boundaries

    Inspect deployment, structured logs, Supabase, provider dashboards, webhooks, queues, jobs, and recent changes without copying secrets or full payloads into the incident record.

  4. 4

    Choose recovery

    Roll back application code only when the prior release is schema-compatible. Use reviewed forward repair for schema and data defects.

  5. 5

    Verify customer journeys

    Check health, auth, billing, webhooks, email, scheduled work, and the exact affected workflow. Monitor queues and projections until they catch up.

  6. 6

    Close with prevention

    Document root cause, impact, recovery, data repair, achieved recovery time, and at least one tested prevention change.

Recovery choices

SituationPreferred actionRequired verification
Application regression without incompatible schemaRoll back to the last verified deployment.Both health endpoints plus auth, billing, email, webhook, and job smoke tests.
Destructive or incompatible database changeCreate a reviewed forward-repair migration or bounded repair script.Affected invariants, generated types, RLS/grants, representative reads/writes, and reconciliation totals.
Provider event processing failureFix the handler or configuration, then replay/retry using the durable provider event identity.No duplicate financial or notification side effects; projected state matches provider state.
Notification backlogRestore scheduler/provider readiness and let atomic claims plus retry policy drain the queue.Queue age falls, attempts are visible, and dead letters receive deliberate operator action.
Data loss or corruptionRestore into an isolated environment first, determine recovery point, then execute the approved recovery plan.Auth-linked profiles, RLS, billing/entitlements, representative storage objects, and customer journeys.
Implementation reference4 areas

These Product code locations explain how the documented behavior is implemented. Expand them when you are ready to customize or maintain this area.

Health

Separate liveness from dependency readiness so deployments and incidents diagnose the right failure.

  • src/proxy.ts
  • src/app/api/health/ready/route.ts

Scheduled work

Publishing, notification delivery, and retention have independently protected entrypoints and services.

  • src/app/api/scheduler/publish/route.ts
  • src/app/api/cron/notification-outbox/route.ts
  • src/app/api/cron/data-retention/route.ts

Webhook processing

Signature validation, durable event processing, and status tracking make provider retries observable.

  • src/lib/security/webhooks/processing.ts
  • src/features/notifications/server/webhooks/resend/handler.ts

Structured observation

Stable identifiers and redaction-aware logging support investigation without exposing credentials.

  • src/lib/observe/logging/logging.ts
  • src/lib/observe/report-error.ts

Reference paths are relative to the Shipflash-Product checkout.

Backup and restore evidence

  • Select a Supabase backup or point-in-time recovery option appropriate to the business promise.
  • Back up Supabase Storage objects separately; database backup does not prove object recovery.
  • Protect recovery access and configuration notes for DNS, deployment, Supabase, billing providers, Resend, schedulers, and monitoring.
  • At least quarterly, restore into an isolated project and verify auth-linked profiles, RLS, billing projections, entitlements, credits, and representative storage objects.
  • Record the achieved recovery point and recovery time; do not advertise an RPO or RTO that has not been demonstrated.
  • Keep one-off repair SQL separate from permanent migrations and require review, dry-run evidence, bounded scope, verification, and an owner.