Incidents

Incidents

Last updated: 2026-07-28 (INC-004 resolved same day)

INC-004 — 2026-07-28 — Severity: P1 — RESOLVED

What happened: svc-auth’s POST /auth/otp/send (consumer phone-OTP login, PR #511) was deployed to production and hit Twilio error 572006 (“Invalid template name”) — the production Twilio account is a trial tier that rejects arbitrary custom SMS text, accepting only predefined template names. A workaround was implemented and deployed: send the literal string 'sms_2fa' as the SMS body instead of real text, and recover the “auto-generated” verification code Twilio substitutes into its own canned message from the API response’s echoed body field, rather than generating a code locally. This was live-verified end-to-end (real SMS delivered, otp/verify and tokens/verify both succeeded) and appeared correct. Follow-up testing (4 separate live calls to the Twilio API) found the “auto-generated” code was a fixed constant (482913) on every single send — not random per request. This made otp/verify a live authentication bypass: any caller able to reach otp/send for a given phone number (any holder of a valid sk_* key with consumer:write/consumer:read scopes — currently only Tap2’s tenant key) could subsequently call otp/verify with the known constant and obtain a valid cs_sess_* session for that phone number’s consumer identity, without ever seeing the SMS or needing possession of the phone. The exposure was compounded by the fixed code being documented in CLAUDE.md, VENDORS.md, and a GitHub PR body while the vulnerable code was live.

Root cause: The workaround was validated by confirming the flow worked (SMS delivered, code extracted, session minted) but not by confirming the extracted code was actually unpredictable — a single successful live test was treated as sufficient evidence for a security property (per-request randomness) that it does not establish. The underlying Twilio behavior (a trial-tier “magic template” returning a static demo message) is undocumented and was discovered empirically, not from Twilio’s published API reference — there was no documented contract to have relied on in the first place.

Detection: A background/automated security review of the commit flagged “Weak Cryptographic Primitives / OTP predictability” shortly after deploy. That prompted a direct re-test (4 live calls to Twilio’s Messages API), which confirmed the code was static, not a false positive.

Resolution: Reverted the workaround in full within the same session, immediately on confirmation — not patched around. consumer-auth.ts’s handleOtpSend was restored to always generate its own cryptographically random code locally (generateOtpCode(), crypto.getRandomValues-backed) and send it as real custom SMS text, exactly matching PR #511’s original design; messaging’s response-body-echo mechanism and the code-extraction helper were removed entirely, not left dormant. Redeployed both svc-auth and messaging to production. Live-verified closed: the previously-valid fixed code no longer authenticates (400 no_pending_challenge); otp/send now correctly fails loud again (502 sms_send_failed — the trial-tier custom-text restriction is unresolved, so this is expected and safe, not a regression). Confirmed via direct production D1 query: zero cs_sess_* session rows exist from the exposure window — no session survived to be abused. Exposure window: approximately 10 minutes (deploy ~14:28:53Z, revert redeployed ~14:38:46Z, 2026-07-28), during which the only credential capable of exploiting it was Tap2’s own sk_live_* tenant key — no evidence of exploitation by any other party, and Tap2’s wallet-api was not yet pointed at these routes in production at the time (see docs/superpowers/plans/2026-07-28-tap2-wallet-production-launch.md).

Prevention: Recorded as a standing invariant, not just a one-off fix — see VENDORS.md’s Twilio entry: the consumer OTP code must always be generated locally (crypto.getRandomValues-backed), never sourced from or echoed by a third-party vendor response, without independently verifying per-request randomness across multiple live calls first. More generally: when a security-relevant value (a code, a token, a secret) originates from a third-party API response rather than being generated locally, “the flow completed successfully” is not sufficient validation — the value’s actual unpredictability must be checked empirically (multiple live samples, not one) before it’s trusted as a credential.

INC-003 — 2026-07-25 — Severity: P1 — RESOLVED (mostly)

What happened: The finstack.sh Cloudflare zone was deleted from the FGV account (confirmed via live API — GET /zones?name=finstack.sh on FGV returns empty; not paused, not transferred, genuinely gone with no undo path). This took down api.finstack.sh, webhooks.finstack.sh (real Stripe webhooks), app.finstack.sh (dashboard), docs.finstack.sh (developer portal), and Paystream’s custom domain simultaneously — full production outage for every public-facing hostname. Root cause: Not established with certainty — the deletion happened outside this session’s own actions (every action taken here up to that point was read-only GETs and one no-op PATCH on a DNS record’s TTL, confirmed via the Cloudflare API audit trail available). Most likely a manual action against the FGV Cloudflare account, possibly in the course of exploring or progressing the separate, already-in-progress FGV→Galactic account migration (TODO.md’s “FGV → Galactic Cloudflare account migration” section) — but this is inference, not confirmed. Resolution: The zone cannot be recreated on FGV in any way that recovers state (DNS records, certs, and history are gone). The only viable path was activating the Galactic-hosted zone that already existed, inertly, from that same in-progress migration’s earlier prep phase (2026-07-18–24). With explicit real-time user go-ahead obtained at the point of action: repointed the registrar’s (Spaceship) nameservers from FGV’s brett/jewel.ns.cloudflare.com to Galactic’s annabel/jaziel.ns.cloudflare.com. Along the way, found and fixed two gaps the original inert-prep phase had missed: app.finstack.sh/docs.finstack.sh had no DNS records at all in the Galactic zone (added both CNAMEs, re-registered the Pages custom domains). Triggered an immediate Cloudflare zone-activation recheck (PUT .../activation_check) rather than waiting for its own polling cycle. 5 of 6 production hostnames confirmed live within about an hour (api., webhooks., app., docs., apex) — the sixth, paystream.api.finstack.sh, needed its own dedicated certificate (the zone-wide wildcard doesn’t cover a 2-level-deep hostname) and was still pending validation as of this writing, ~3h later; not urgent, since Paystream has zero live transactions and every capability on that subdomain is identically available via plain api.finstack.sh. Side effect, not fully resolved: This incident recovery is, functionally, the FGV→Galactic Rust account migration completing — but via emergency path, not the planned one. It skipped planned step 6 (updating Rust’s hardcoded custom_domains.rs CF_ACCOUNT_ID/CF_ZONE_ID constants + the finstack-api Fly secret CF_API_TOKEN) and the admin-gateway WAF allowlist rule. The Rust constants now point at the deleted FGV zone — a live bug (see BUGS.md), not just stale — blocked on the same missing Fly.io credentials that blocked this step before the incident. See TODO.md’s migration section for full current status. Prevention: No monitoring/alerting exists today that would have caught a zone deletion before a customer-facing outage did — worth a follow-up (a scheduled GET /zones?name=finstack.sh health check comparing zone existence + status against an expected account, alerting on mismatch, would have caught this in seconds instead of relying on someone noticing the outage).

INC-002 — 2026-07-16 — Severity: P3 — RESOLVED

What happened: finstack-cli migrate status against prod (Neon ancient-heart-91326215) showed migrations 0073–0076 (locations/merchant fees, the card-present location CHECK validation, merchant-scoped processor routing, and this session’s terminal_applications) as pending — merged to main across several PRs over the preceding days but never run against prod. Unlike INC-001, this was pure lag, not drift: no _sqlx_migrations gaps, no untracked-but-applied objects — the migrations simply hadn’t been executed since merge. No user-facing incident triggered discovery; caught while shipping PR #445 and checking whether its own migration needed applying. Root cause: Same root cause INC-001’s “Prevention” note anticipated and flagged for monitoring: no periodic drift/lag check exists between main’s migrations/ directory and prod’s _sqlx_migrations max version, so a merged-but-unrun migration has no alerting path — it’s silent until something that depends on the new schema fails at runtime. Resolution: Ran finstack-cli migrate run against DATABASE_ADMIN_URL (via Neon MCP owner connection string). All 4 pending migrations applied cleanly in one pass; migrate status now shows 76/76 applied, no gaps. Confirmed live: terminal_applications has relrowsecurity/relforcerowsecurity both true, matching the migration’s design. Prevention: Built same day — finstack-cli migrate status --check (exits non-zero if any migration is pending) plus .github/workflows/migration-drift-check.yml (daily cron + workflow_dispatch, read-only against DATABASE_URL). See RUNBOOK.md’s “Automated drift/lag check”. Not yet active: the workflow needs a DATABASE_URL GitHub Actions secret added manually (credential only a repo admin can provide) — until then it fails closed with a connection error, not a real signal.

INC-001 — 2026-07-13 — Severity: P1 — RESOLVED

What happened: Prod’s _sqlx_migrations tracking table (Neon ancient-heart-91326215, the same database both DATABASE_URL (pooled) and DATABASE_ADMIN_URL (direct) connect to) tops out at version 50 (2026-06-14), but migrations 51–69 were found fully applied in the live schema anyway — untracked, applied out-of-band without going through finstack-cli migrate run. Migrations 0052 (billing_plans/primitive-billing), 0053 (ai_tasks/primitive-ai), 0070 (moov_customer_accounts), and 0071 (refunds.attempt_count) were never applied at all. Discovered when an unrelated migration (0072, dropping payfac_sub_merchants) failed at version 51 with relation "custom_domains" already exists. Confirmed production impact: Moov CNP card-present payments (moov_customer_accounts missing — MoovProcessor::create_customer_account, POST /v1/payments/moov/drop-session) and the Moov refund reconciler (refunds.attempt_count missing — moov_refund_reconciler.rs) are non-functional in production, contradicting CLAUDE.md’s claim of a live-verified $10 Moov transaction. primitive-billing and primitive-ai are also fully non-functional (zero tables exist for either). Root cause: Migrations 51–69 were applied to production by some path other than finstack-cli migrate run (direct psql, a one-off script, or a different tool) without the migrator’s bookkeeping (_sqlx_migrations) ever being updated to match. No success=false/dirty rows exist for these versions — the tracking table simply has no record of them at all, despite the objects existing and matching their migration files exactly (clean, all-or-nothing per file — no partial applications found across 51–69). Resolution: Baselined _sqlx_migrations for the 17 migrations (0051, 0054–0069) confirmed fully applied but untracked, using checksums computed by sqlx’s own Migrator (validated byte-for-byte against 3 known-good stored checksums for versions 1/19/50 before use — not hand-derived). Ran finstack-cli migrate run against the direct Neon endpoint, which applied the 4 genuinely-missing migrations (0052 billing, 0053 AI primitive, 0070 Moov customer accounts, 0071 refunds attempt_count) plus 0072 in sequence. migrate status now shows all 72 versions applied with no gaps. Confirmed live: billing_plans, ai_tasks, moov_customer_accounts exist, refunds.attempt_count exists, payfac_sub_merchants is gone. Moov CNP card payments and the Moov refund reconciler are unblocked. Full per-migration drift map and remediation trace: .superpowers/sdd/migration-drift-report.md (session-local, not committed — summary captured here and in the runbook). Prevention: All schema changes must go through finstack-cli migrate run against DATABASE_ADMIN_URL — no direct psql/manual DDL against production, ever. Consider a periodic drift-detection check (compare _sqlx_migrations max version against ls migrations/ | tail -1) as a monitoring signal.