Skip to content
FinStack Docs

Runbook

Runbook

Section titled “Runbook”

Last updated: 2026-07-25 — ADR-9: finstack-rs (Rust/Fly.io/Postgres) fully deleted. Every procedure below this notice was written for that deleted backend — Postgres role provisioning, Fly health checks, Prometheus/Alertmanager monitoring, and the Rust-specific incident/migration-drift procedures no longer apply to anything running. They’re kept as historical/reference material, not as current operational procedure. A finstack runbook (D1 migration recovery, wrangler tail-based health checks, CF Workers incident response) does not exist yet — this is new-content authoring beyond the scope of this removal pass; tracked in TODO.md.

DB Role Provisioning (historical — finstack-rs, deleted)

Section titled “DB Role Provisioning (historical — finstack-rs, deleted)”

The Rust backend requires two Postgres roles with specific RLS attributes. Run this once per environment after creating the Postgres database.

Dev (docker compose):

Terminal window
cd finstack-rs
bash scripts/provision_db.sh
# Uses POSTGRES_USER=finstack, default dev passwords
# Runs RLS leak-check at end — should return 0 leaked rows

Prod (Neon / Fly Postgres / RDS):

Terminal window
export DATABASE_ADMIN_URL="postgres://owner:pw@host/finstack"
export FINSTACK_APP_PASSWORD="$(openssl rand -base64 32 | tr -d /=+ | head -c 40)"
export FINSTACK_WORKER_PASSWORD="$(openssl rand -base64 32 | tr -d /=+ | head -c 40)"
cd finstack-rs
bash scripts/provision_db_prod.sh
# Store both passwords in Fly secrets / 1Password before the shell closes
# fly secrets set DATABASE_URL_APP=postgres://finstack_app:$FINSTACK_APP_PASSWORD@host/finstack
# fly secrets set DATABASE_URL_WORKER=postgres://finstack_worker:$FINSTACK_WORKER_PASSWORD@host/finstack

Role model:

RoleBYPASSRLSUsed by
finstack_appNOAPI server — all queries RLS-scoped to app.tenant_id
finstack_workerYESBackground worker — webhook sweep, analytics rollup, cross-tenant reads

Note (Neon): Neon’s free tier does not support BYPASSRLS. The finstack_worker role will be created but the BYPASSRLS attribute will be silently stripped. The provisioning script’s role-attribute check will print MISCONFIGURED — upgrade to a paid Neon project or use SET LOCAL row_security = off requires a superuser role workaround. See crates/finstack-webhooks/src/service/sweep.rs for the impact.

Common Operations

Section titled “Common Operations”

Deploy

Section titled “Deploy”
Terminal window
galactic deploy --slug finstack

Tail Logs

Section titled “Tail Logs”
Terminal window
galactic logs finstack

Query Database

Section titled “Query Database”
Terminal window
galactic db query "SELECT ..." --slug finstack

Health Checks

Section titled “Health Checks”
  • GET /healthz → {"status":"ok"} — checked by Fly.io every 10s; gates deploys
  • Prometheus metrics: http://finstack-api.internal:9090/metrics — internal Fly network only

Monitoring Setup

Section titled “Monitoring Setup”

Metrics collection

Section titled “Metrics collection”

Fly.io scrapes finstack-api Prometheus metrics automatically via the [metrics] block in finstack-rs/fly.toml (port 9090, path /metrics). No separate Prometheus deployment needed.

View metrics at https://fly-metrics.net → select finstack-api.

Currently emitted (crates/api-rest/src/metrics_middleware.rs):

  • http_requests_total{method, route, status} — HTTP requests by status code
  • http_request_duration_seconds{method, route} — request latency histogram

Planned — pending EPIC-011 instrumentation (alert rules exist; metrics not yet emitted):

  • outbox_pending_total — events queued for webhook fan-out (→ OutboxBacklogHigh)
  • webhook_deliveries_total{status} — delivery attempts by outcome (→ WebhookDeliveryFailureHigh)
  • payment_transitions_total{to} — payment FSM transitions (→ PaymentFailureRateHigh)
  • db_pool_wait_time_bucket — sqlx pool wait histogram (→ DBPoolSaturation)

Alert rules

Section titled “Alert rules”

Alert rules live at docs/prometheus/alerts.yml. To load them in Fly Grafana:

  1. Open https://fly-metrics.net → Alerting → Alert rules → Import
  2. Paste contents of docs/prometheus/alerts.yml

To validate rules locally:

Terminal window
# Install promtool (part of Prometheus distribution)
promtool check rules docs/prometheus/alerts.yml

Alertmanager (notification routing)

Section titled “Alertmanager (notification routing)”

docs/prometheus/alertmanager.yml is an envsubst template — ${VAR} placeholders must be rendered before Alertmanager can load it (Alertmanager does not expand env vars natively).

Locally (testing):

Terminal window
export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
export SLACK_CRITICAL_WEBHOOK_URL=$SLACK_WEBHOOK_URL  # or a separate #critical channel webhook
envsubst < docs/prometheus/alertmanager.yml | alertmanager --config.file=/dev/stdin

Production (Fly.io deploy):

Terminal window
# Set secrets on the alertmanager Fly app
fly secrets set SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... --app finstack-alertmanager
fly secrets set SLACK_CRITICAL_WEBHOOK_URL=https://hooks.slack.com/services/... --app finstack-alertmanager


# Render template and pass to alertmanager at container startup (add to Dockerfile CMD or entrypoint)
envsubst < /app/alertmanager.yml > /etc/alertmanager/alertmanager.yml
alertmanager --config.file=/etc/alertmanager/alertmanager.yml

Local metrics testing

Section titled “Local metrics testing”
Terminal window
# Start the API locally (metrics on port 9090)
cd finstack-rs && cargo run --bin finstack-api


# Confirm metrics endpoint
curl http://localhost:9090/metrics | head -30


# Run Prometheus against local instance
prometheus --config.file=docs/prometheus/prometheus.yml

Sentry error tracking

Section titled “Sentry error tracking”

Sentry is wired via SENTRY_DSN fly secret. Errors at tracing::Level::ERROR are automatically forwarded. Configure in ENVIRONMENTS.md and set via:

Terminal window
fly secrets set SENTRY_DSN=https://... --app finstack-api

Alerting Runbook

Section titled “Alerting Runbook”

OutboxBacklogHigh

Section titled “OutboxBacklogHigh”

Symptom: outbox_pending_total > 100 for 5 minutes. Webhook fan-out is stalling — events are being inserted into the outbox faster than the sweep worker delivers them.

Likely causes:

  1. finstack-worker is down or crashed — galactic logs finstack-worker for error
  2. Webhook endpoints are returning 4xx/5xx, causing retries to pile up
  3. sweeper_claimed_at lock contention from a stuck sweep batch (check for rows with claimed_at older than sweep timeout)

Remediation:

Terminal window
# Check worker health
galactic logs finstack-worker


# Inspect backlog
galactic db query "SELECT endpoint_id, count(*), min(created_at) FROM webhook_outbox WHERE delivered_at IS NULL GROUP BY endpoint_id ORDER BY count DESC LIMIT 10" --slug finstack


# Force-expire stuck locks (sweeper will re-claim)
galactic db query "UPDATE webhook_outbox SET sweeper_claimed_at = NULL WHERE sweeper_claimed_at < NOW() - INTERVAL '5 minutes' AND delivered_at IS NULL" --slug finstack


# If worker down, redeploy
galactic deploy --slug finstack-worker

WebhookDeliveryFailureHigh

Section titled “WebhookDeliveryFailureHigh”

Symptom: rate(webhook_deliveries_total{status="failed"}[10m]) / rate(webhook_deliveries_total[10m]) > 0.05 for 10 minutes. More than 5% of attempted webhook deliveries are failing.

Likely causes:

  1. Customer endpoint is unreachable (DNS failure, TLS cert expired, firewall)
  2. Customer endpoint returning 4xx (auth mismatch, payload schema change)
  3. FinStack misconfigured signing secret

Remediation:

Terminal window
# Find which endpoints are failing
galactic db query "SELECT we.url, wdl.status_code, count(*) FROM webhook_delivery_log wdl JOIN webhook_endpoints we ON we.id = wdl.endpoint_id WHERE wdl.created_at > NOW() - INTERVAL '1 hour' AND wdl.status_code NOT BETWEEN 200 AND 299 GROUP BY we.url, wdl.status_code ORDER BY count DESC LIMIT 20" --slug finstack


# Check if failure is tenant-wide or specific endpoint
# If specific endpoint: notify tenant; disable if >24h persistent
galactic db query "UPDATE webhook_endpoints SET enabled = false WHERE id = '<id>'" --slug finstack


# If signing key mismatch: tenant must rotate via API
# Never update webhook_secret directly in DB

PaymentFailureRateHigh

Section titled “PaymentFailureRateHigh”

Symptom: rate(payment_transitions_total{to="failed"}[5m]) / rate(payment_transitions_total[5m]) > 0.02 for 5 minutes. More than 2% of payment FSM transitions are to the failed state.

Likely causes:

  1. Processor (Stripe/Adyen) degradation or outage — check processor status pages
  2. Card decline surge (fraud wave, BIN range issue)
  3. FinStack ProcessorRouter selecting a misconfigured processor_config

Remediation:

Terminal window
# Check processor error distribution
galactic db query "SELECT processor_error_code, count(*) FROM payments WHERE status = 'failed' AND updated_at > NOW() - INTERVAL '30 minutes' GROUP BY processor_error_code ORDER BY count DESC LIMIT 10" --slug finstack


# Check if failures are on one processor_config
galactic db query "SELECT pc.label, count(*) FROM payments p JOIN processor_configs pc ON pc.id = p.processor_config_id WHERE p.status = 'failed' AND p.updated_at > NOW() - INTERVAL '30 minutes' GROUP BY pc.label" --slug finstack


# If processor degraded: disable that processor_config to stop routing to it
galactic db query "UPDATE processor_configs SET enabled = false WHERE label = '<label>'" --slug finstack


# Monitor Stripe: https://status.stripe.com  |  Adyen: https://status.adyen.com

DBPoolSaturation

Section titled “DBPoolSaturation”

Symptom: histogram_quantile(0.95, rate(db_pool_wait_time_bucket[5m])) > 0.1 for 5 minutes. P95 connection pool wait time exceeds 100ms — queries are queuing for connections.

Likely causes:

  1. Traffic spike exhausting the 5-connection PgBouncer limit per Fly machine
  2. Long-running transaction holding a connection (RLS SET LOCAL in a hung txn)
  3. Neon serverless cold-start latency spike

Remediation:

Terminal window
# Check active connections at Neon
galactic db query "SELECT count(*), state, wait_event_type, wait_event FROM pg_stat_activity WHERE datname = current_database() GROUP BY state, wait_event_type, wait_event" --slug finstack


# Identify long-running transactions
galactic db query "SELECT pid, now() - xact_start AS duration, query FROM pg_stat_activity WHERE xact_start IS NOT NULL AND state != 'idle' ORDER BY duration DESC LIMIT 10" --slug finstack


# Terminate if needed
galactic db query "SELECT pg_terminate_backend(<pid>)" --slug finstack


# If sustained: scale Fly machines (increases pool × machines)
fly scale count 3 --app finstack-api


# If Neon: check branch compute status in Neon console; consider upgrading plan for higher connection limits

APIErrorRateHigh

Section titled “APIErrorRateHigh”

Symptom: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01 for 5 minutes. More than 1% of API requests are returning 5xx errors.

Likely causes:

  1. Panic in a handler — check Sentry for error clustering
  2. Database connectivity issue (Neon unreachable, pool exhausted)
  3. Deployment regression — compare error rate before/after last deploy

Remediation:

Terminal window
# Tail live logs for panic/error messages
galactic logs finstack


# Check Sentry for error clustering (most frequent error)
# https://sentry.io → FinStack project → Issues → sort by frequency


# Rollback if regression confirmed
fly releases --app finstack-api          # list versions
fly deploy --image registry.fly.io/finstack-api:<prev-version> --app finstack-api


# Verify healthz after rollback
curl https://finstack-api.fly.dev/healthz

Incident Response

Section titled “Incident Response”
  1. Page via on-call rotation (see Escalation)
  2. Identify affected surface from alert label (route, method)
  3. Follow per-alert runbook above
  4. Document in INCIDENTS.md after resolution

Escalation

Section titled “Escalation”

Migration Drift Recovery (INC-001)

Section titled “Migration Drift Recovery (INC-001)”

Symptom: finstack-cli migrate run fails with relation "X" already exists partway through, even though _sqlx_migrations shows an earlier max version than the migrations directory.

Cause: One or more migrations were applied to prod outside finstack-cli migrate run (direct psql, a one-off script) without the _sqlx_migrations bookkeeping table being updated to match.

Diagnosis (read-only, safe to run anytime):

  1. SELECT * FROM _sqlx_migrations ORDER BY version; — find the real max tracked version.
  2. For every migration file after that version, read its DDL and check via information_schema.tables/information_schema.columns/pg_indexes whether its objects already exist live.
  3. Categorize each as FULLY APPLIED (all objects exist), PARTIALLY APPLIED (some exist), or NOT APPLIED (none exist).

Recovery (only after diagnosis confirms which category each migration falls into):

  1. Do not just re-run migrate run — it stops at the first already-applied migration it reaches.
  2. For every FULLY APPLIED migration missing from _sqlx_migrations: insert a row with the correct version, description, success = true, and a checksum computed by sqlx itself (not hand-derived) — write a tiny throwaway program using sqlx::migrate::Migrator::new(source) to enumerate the real Migration structs (each carries its own correctly-computed checksum), and use those exact bytes in the INSERT. A wrong or hand-guessed checksum algorithm will cause sqlx to reject the row or silently diverge from what a real migrate run would have recorded.
  3. Any PARTIALLY APPLIED migration needs manual reconciliation first (apply just the missing objects by hand, matching the migration file exactly) before it can be baselined as applied — do not baseline a partial migration as if it were complete.
  4. Once every already-applied migration is correctly baselined, run finstack-cli migrate run normally — it will apply only the genuinely NOT APPLIED migrations from here forward.
  5. Verify: _sqlx_migrations max version matches the latest file in migrations/, and spot-check the previously-missing tables/columns now exist.

INC-001 specifics (2026-07-13): migrations 51–69 were FULLY APPLIED but untracked (baselined); 52 (billing_plans), 53 (ai_tasks), 70 (moov_customer_accounts), 71 (refunds.attempt_count) were NOT APPLIED (applied normally after baselining). See INCIDENTS.md INC-001.

INC-002 specifics (2026-07-16): a milder recurrence — pure lag, not drift. Migrations 73–76 were fully tracked-and-correct in every migration file, just never run against prod after merging to main. No baselining needed; finstack-cli migrate run applied all 4 cleanly in one pass. See INCIDENTS.md INC-002.

Automated drift/lag check

Section titled “Automated drift/lag check”

.github/workflows/migration-drift-check.yml runs finstack-cli migrate status --check against prod daily (and on-demand via workflow_dispatch) — exits non-zero (failing the job) if any migration in migrations/ isn’t yet reflected in prod’s _sqlx_migrations, which is exactly the gap that caused both INC-001 and INC-002. Read-only: the check only SELECTs from _sqlx_migrations, and finstack_app has a SELECT grant on it (has_table_privilege('finstack_app', 'public._sqlx_migrations', 'SELECT') confirmed true against prod, 2026-07-16 — don’t assume this from “no RLS on the table” alone, that’s a different mechanism than a grant), so the finstack_app-role DATABASE_URL is sufficient; no DATABASE_ADMIN_URL needed.

migrate status/--check distinguishes three states per migration: applied, pending (in migrations/ but not yet run — the INC-001/INC-002 pattern), and DIRTY (a row exists with success=false — the migration started applying and failed partway through; migrate run itself refuses to proceed at all until this is resolved manually, see “Migration Drift Recovery” above). --check fails on either pending or dirty. A genuinely never-migrated database (no _sqlx_migrations table yet) reports every migration as pending rather than erroring.

One-time setup required (not done by this change): add a DATABASE_URL repo secret in GitHub → Settings → Secrets and variables → Actions, using the pooled finstack_app-role connection string (see ENVIRONMENTS.md — never the owner/DATABASE_ADMIN_URL credential; this check never writes). Until that secret exists, the workflow’s preflight step fails immediately with a distinctly-titled Migration Drift Check Not Configured error — visible in the Actions run list without needing to open the log — rather than looking identical to a real drift alert.

Paystream Migration Cutover (EPIC-017 TICKET-17.5)

Section titled “Paystream Migration Cutover (EPIC-017 TICKET-17.5)”

Paystream runs natively on FinStack — no compatibility shim. The old D1/Hono stack is decommissioned after DNS cutover. This runbook covers the full migration sequence.

Pre-conditions

Section titled “Pre-conditions”
  • EPIC-013 through EPIC-016 deployed (payout, recipient, funding, routing primitives live)
  • Migration 0051 applied (custom_domains table)
  • CF_API_TOKEN, RECIPIENT_BANK_MASTER_KEY, FINSTACK_ADMIN_TOKEN set as Fly secrets
  • PAYSTREAM_CF_ACCOUNT_ID, PAYSTREAM_D1_DATABASE_ID, PAYSTREAM_CF_API_TOKEN available locally

Step 1 — Export Paystream D1 data

Section titled “Step 1 — Export Paystream D1 data”
Terminal window
PAYSTREAM_CF_ACCOUNT_ID=<id> \
PAYSTREAM_D1_DATABASE_ID=<id> \
PAYSTREAM_CF_API_TOKEN=<token> \
npx tsx scripts/paystream-d1-export.ts
# Creates: exports/paystream-{contacts,funding_sources,transfers,balances}-{ts}.ndjson

Dry-run first: add --dry-run to count rows without writing files.

Step 2 — Provision Paystream tenant + sk_* key

Section titled “Step 2 — Provision Paystream tenant + sk_* key”
Terminal window
FINSTACK_ADMIN_TOKEN=<token> \
PAYSTREAM_USER_ID=paystream_prod \
bash scripts/provision-paystream-keys.sh

Save the returned sk_live_paystrm*_* key and tenant UUID immediately — the secret is shown only once (argon2id hash stored; original lost).

Record:

  • PAYSTREAM_TENANT_ID = tenant UUID from response
  • PAYSTREAM_SK_KEY = sk_live_paystrm*_* from response

Step 3 — Import data to Neon

Section titled “Step 3 — Import data to Neon”
Terminal window
DATABASE_WORKER_URL=<neon-bypassrls-url> \
PAYSTREAM_TENANT_ID=<uuid-from-step-2> \
RECIPIENT_BANK_MASTER_KEY=<base64-32-bytes> \
npx tsx scripts/paystream-import.ts

Dry-run first: add --dry-run to validate files.

Expected output:

  recipients:     N inserted  0 updated  0 skipped
  bank accounts:  N inserted  0 updated  0 skipped  (verification_status=unverified)
  payouts:        N inserted  0 updated  0 skipped
  balance:        1 inserted  0 updated  0 skipped

Import is idempotent — re-running on failure is safe.

Step 4 — Register custom domain paystream.api.finstack.sh

Section titled “Step 4 — Register custom domain paystream.api.finstack.sh”
Terminal window
curl -X POST https://api.finstack.sh/v1/custom-domains \
  -H "Authorization: Bearer ${PAYSTREAM_SK_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"slug": "paystream"}'
# Response: {"id": "...", "dns_verified": true}
# Slug subdomains are auto-verified at creation.

Verify slug is live:

Terminal window
curl https://paystream.api.finstack.sh/healthz
# Expected: {"status":"ok"}

Step 5 — Update Paystream client config (parallel with DNS prep)

Section titled “Step 5 — Update Paystream client config (parallel with DNS prep)”

Paystream operators update their config:

FINSTACK_API_KEY=sk_live_paystrm*_*
FINSTACK_API_URL=https://paystream.api.finstack.sh

Smoke test before DNS cutover (clients talking to FinStack):

Terminal window
curl -X GET https://paystream.api.finstack.sh/v1/recipients \
  -H "Authorization: Bearer ${PAYSTREAM_SK_KEY}"
# Expected: 200 {"recipients": [...]}

Step 6 — Reduce DNS TTL (48 hours before cutover)

Section titled “Step 6 — Reduce DNS TTL (48 hours before cutover)”

In Spaceship/Cloudflare DNS for paystream.fi:

api.paystream.fi  CNAME  →  <current-paystream-hono-worker>.workers.dev  TTL 60s

Confirm TTL propagated:

Terminal window
dig api.paystream.fi +short

Step 7 — DNS cutover

Section titled “Step 7 — DNS cutover”

Set TXT verification record first:

Terminal window
# In Spaceship DNS for paystream.fi:
_finstack-verify.api.paystream.fi  TXT  "finstack-domain-verification=<token-from-POST-response>"

Register custom domain in FinStack:

Terminal window
curl -X POST https://paystream.api.finstack.sh/v1/custom-domains \
  -H "Authorization: Bearer ${PAYSTREAM_SK_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"slug": "paystream", "domain": "api.paystream.fi"}'
# Note: CF for SaaS requires ssl_and_certificates:edit on FGV CF token (deferred)
# Until that's set, custom domain provisioning returns 500 for arbitrary domains.
# Use slug subdomain for cutover instead.

Update CNAME (slug subdomain path — works today):

api.paystream.fi  CNAME  →  paystream.api.finstack.sh  TTL 60s

Trigger TXT verification:

Terminal window
curl -X POST https://paystream.api.finstack.sh/v1/custom-domains/<domain-id>/verify \
  -H "Authorization: Bearer ${PAYSTREAM_SK_KEY}"
# Expected: {"dns_verified": true}

Step 8 — Monitor 24h post-cutover

Section titled “Step 8 — Monitor 24h post-cutover”

Success metrics:

  • Error rate < 0.1% for 24 hours post-cutover
  • P99 latency < 500ms
  • No payout failures in Sentry
Terminal window
# Tail FinStack logs
galactic logs finstack


# Check error rate
galactic db query "SELECT status, count(*) FROM payouts WHERE tenant_id='<paystream-tenant-id>' AND created_at > now() - interval '1 hour' GROUP BY status" --slug finstack

Step 9 — Decommission legacy Hono worker

Section titled “Step 9 — Decommission legacy Hono worker”

Only after 24h stable:

Terminal window
wrangler delete --name paystream-api  # in Paystream's CF account
# Or simply remove the route/trigger — keep the worker for 7 days as fallback

Rollback plan (any step before Step 9):

Terminal window
# Point api.paystream.fi back to legacy Hono worker:
api.paystream.fi  CNAME  →  paystream-api.<cf-account>.workers.dev
# DNS propagates in ≤ TTL seconds (60s after Step 6)
# Neon data is non-destructive — nothing to undo on import

Post-Migration Reference

Section titled “Post-Migration Reference”

Tenant: Provisioned in Step 2 (run provision-paystream-keys.sh to get UUID) Slug subdomain: paystream.api.finstack.sh (live after Step 4) Custom domain: api.paystream.fi (CNAME → paystream.api.finstack.sh, live after Step 7) Scopes: payouts:read, payouts:write, recipients:read, recipients:write, platform:read, messaging:write, domains:read, domains:write

Key endpoints:

  • POST /v1/recipients — create payee
  • POST /v1/payouts — initiate disbursement
  • GET /v1/platform/accounts/usd — check balance
  • POST /v1/auth/otp/send + /verify — consumer OTP auth
  • POST /v1/custom-domains — register slug or custom domain

Bank account re-link notice: imported bank accounts have verification_status=unverified. Recipients must re-add their bank accounts via Paystream’s UI to trigger micro-deposit verification before new payouts can process. Existing historical payout records are intact for audit.