Trust Center

Business Continuity and Disaster Recovery Plan

How we keep the service available during disruption — and how we recover when something breaks.

Version: 1.4Effective: 2026-10-26Owner: Howell & Gibbs LLCDownload PDF

This document describes how SendTax keeps the service available during disruption, what we do when a disruption becomes a disaster, and how we recover. It is a companion to our Information Security, Incident Response, and Vendor Management policies.

We are publishing this plan as a candid description of our current posture at the company's current stage. We are not claiming multi-region active-active deployment, validated recovery-time objectives, or capabilities we have not built. Items marked Roadmap are deliberate next steps.


1. Scope

This plan covers:

  • The technical resilience of the SendTax service.
  • The procedures we follow when a component fails or a region becomes unavailable.
  • Our preparations for events that affect the SendTax operating team itself (key-person unavailability, loss of operator device).

It does not cover financial business continuity for Howell & Gibbs LLC as a company; that is addressed separately. Incidents that affect security but not availability are governed by the Incident Response Policy.


2. Definitions

TermMeaning
RTORecovery Time Objective. The maximum acceptable duration of an outage before the service is restored.
RPORecovery Point Objective. The maximum acceptable amount of data loss, measured backward from the moment of failure.
BIABusiness Impact Analysis. The prioritization of services and data by their criticality.
Single point of failure (SPoF)A component whose loss takes down the service even if redundant components elsewhere are healthy.
Hot standbyA redundant instance running in parallel and able to take over with minimal delay.
Cold restoreA recovery procedure that rebuilds a service from backups, with measurable downtime.
PITRPoint-in-time recovery. Restoring a database to a chosen moment within the backup retention window.

3. Business Impact Analysis

In recovery, components are restored in this priority order:

PriorityComponentWhy it ranks here
P1Database (Fly.io Managed Postgres) and document storage (Cloudflare R2)No other component is useful without customer data being readable.
P1Google Cloud KMSDocuments cannot be decrypted without KMS unwrap; this gates all customer-document operations.
P1Clerk (authentication)Customers cannot reach their data without sign-in.
P2API service (st-api-prod)Required for any customer interaction beyond cached pages.
P2Celery worker (st-worker-prod)Required for document classification, OCR, scheduled obligations, and email fan-out — the service is degraded but partially usable without it.
P3Web apps (the send.tax marketing and sign-in site on Vercel; the filer, preparer and admin apps on Fly.io)Browser-side concerns. Static content can be served from cache during a backend outage; signed-in interactions cannot.
P3ResendOutbound transactional email. Account-critical flows degrade; data remains intact.
P3Sentry, PostHog, ModalObservability and ML inference. The service runs without them. If Modal is unavailable, the worker retries briefly, then guesses the document's type from its file name and flags it for review; the document is not sent to any other model or service.

4. Resilience controls currently in place

This section describes what is actually deployed today.

4.1 Application layer (Fly.io)

  • Primary region: All SendTax application machines on Fly.io (API, workers, and the filer, preparer and admin web apps) run in Fly.io's iad region (US-East, Ashburn, Virginia). This is configured explicitly in every fly.toml in the repo. The marketing and sign-in site (send.tax) is hosted on Vercel.
  • Per-machine resilience: Fly restarts a machine whose process crashes; workers have [[restart]] policy = "always" with up to 5 retries. A failing health check does not restart a machine: it marks the machine unhealthy and blocks a deploy. The API service and the filer and preparer web apps are configured with min_machines_running = 2 and auto_start_machines = true — so a single machine failure leaves a live peer serving traffic.
  • Health checks: /health/ on the API and a broker-connectivity sidecar on the worker, both polled every 30 seconds by Fly.
  • Deploy gate: Failed health checks block deployments and surface in fly checks list.

4.2 Database layer (Fly.io Managed Postgres)

  • Managed service: Production uses Fly.io Managed Postgres (st-pg-prod), which provides automatic failover and managed high availability per Fly's published terms.
  • Backups: Automated daily backups with point-in-time recovery within the managed plan's retention window. We do not maintain a separate backup pipeline.
  • Encryption: Storage and backups are encrypted at rest by Fly's managed service.

4.3 Document storage (Cloudflare R2)

  • Durability: R2 provides Cloudflare's published durability guarantees; our bucket was created in Cloudflare's us jurisdiction, so its objects stay in the United States.
  • Encryption layers: Uploaded documents are AES-256-GCM-encrypted by the SendTax application as soon as they arrive, before processing (see Encryption Policy § 7.1); R2 also applies its own at-rest encryption — so customer documents are encrypted twice, with SendTax holding the key-management end of the outer layer (see Encryption Policy).

4.4 Key management (Google Cloud KMS)

  • Managed service: KEK custody is in GCP Cloud KMS, region us-east1. KMS is operated by Google and inherits its SLA.
  • Authentication: Our provisioning sets the application servers to authenticate with Workload Identity Federation, which uses no long-lived service-account key. The software also supports a service-account key, and we have not yet confirmed which of the two production uses.
  • Dual-key migration path: The encryption module supports pointing wrap operations at a new KEK while still being able to unwrap DEKs that were wrapped under a legacy KEK — usable as a rolling re-encryption path in a key compromise.

4.5 Identity (Clerk)

  • Managed service: Authentication is delegated to Clerk, which operates its own multi-region redundancy. We inherit Clerk's availability for sign-in flows.

4.6 Operational resilience

  • Infrastructure as code: Fly configurations are committed to the monorepo (fly.api.prod.toml, fly.worker.prod.toml, and the per-app fly.toml files). The cluster can be re-provisioned from version control.
  • Secrets recovery: Application secrets are stored both in Fly encrypted secrets and (operator-accessible) in environment-scoped 1Password vaults, so the loss of either does not prevent rebuilding the cluster.

5. Single points of failure (and how we handle them)

We deliberately enumerate these rather than hide them.

SPoFRiskCurrent mitigation
Fly.io region iadA region-wide Fly outage takes down all SendTax application services simultaneously.We accept this risk at our current stage. Mitigation is reactive: rebuild in an alternate Fly region from infrastructure-as-code and managed-Postgres restore if iad is unrecoverable for an extended period.
Celery Beat schedulerExactly one Beat instance runs; periodic tasks (purge sweeps, scheduled obligations) stop if it crashes.[[restart]] policy = "always" auto-restarts Beat, and a watchdog task checks in with a Sentry cron monitor every 5 minutes; a missed check-in sends an email alert to the operators. It does not page anyone's phone yet. (Roadmap: page the on-call by phone.) Beat crashes affect scheduling, not customer-facing reads.
Single application database clusterDatabase corruption — not just outage — would affect the live database.Daily backups + PITR per § 4.2; backups are managed-service-encrypted and stored by Fly.
Cloudflare R2 bucketA bucket-level configuration error or compromised credential could affect document availability.Application-layer encryption means a compromised bucket does not expose document plaintext. R2 offers no object versioning, and we keep no separate copy of R2 objects: the bucket is the one copy of each document. A plain copy would keep each document from before we encrypted it (Encryption Policy § 7.1) and every deleted document, which would defeat our encryption and deletion. We accept that a document deleted by mistake, or lost to a compromised storage credential, can't be restored from storage; the customer uploads it again. A copy of encrypted objects only, in a second account under a short lock, is under consideration. (Roadmap)
GCP KMS key version availabilityIf a KEK version becomes unavailable (deletion, account compromise, GCP outage), the documents wrapped under that version cannot be decrypted until access is restored.Dual-key support allows rotation without losing access to previously-wrapped DEKs. We do not destroy KEK versions outside of documented data-deletion workflows.
Key-person availabilityA two-person operating team is sensitive to individual unavailability during an incident.A documented break-glass procedure is how the other operator deploys a production fix when the Qualified Individual is unavailable; its credentials have not yet been tested (§ 6.10). Runbooks cover the most common recovery scenarios. (Roadmap: add an external operator on retainer for stronger business-continuity coverage as the company grows.)

6. Threat scenarios and recovery procedures

We organize procedures by failure scope. Each procedure is the intended action; the Incident Response Policy governs the broader response (declaration, communication, postmortem).

6.1 Single machine failure

Scope: One Fly machine fails (process crash, VM death). Detection: Fly health checks fail; Sentry may surface symptoms. Procedure: Fly restarts a crashed process. If a machine dies, the API and the filer and preparer apps keep serving from their second machine. No operator action is required in most cases. If restarts loop, an operator investigates and may roll back the most recent deployment. Expected recovery: Seconds to minutes.

6.2 Application deployment failure

Scope: A bad deploy makes the API or worker unhealthy. Detection: Failed health checks block deployment; Sentry surfaces errors post-deploy if a subtle regression slips through. Procedure: Roll back to the previous image tag (fly deploy --image <previous-tag>). Investigate after rollback; do not debug in production. Expected recovery: Minutes.

6.3 Database outage (within Fly.io managed service)

Scope: Managed Postgres becomes unavailable but the cluster is still under Fly's control. Detection: API health checks fail with database errors; Sentry surfaces connection failures. Procedure: Engage Fly.io support immediately. Fly's managed service is responsible for failover; we monitor and communicate. Expected recovery: Per Fly Managed Postgres SLA.

6.4 Data corruption requiring restore

Scope: Logical corruption (a bad migration, application bug writing wrong data) where the live database is intact but contains incorrect data. Procedure:

  1. Stop write-path workers immediately (fly scale count 0 on st-worker-prod).
  2. Determine the corruption window (when did the bad writes begin?).
  3. Engage Fly support to restore from PITR to the latest pre-corruption point.
  4. Reconcile any legitimate writes that occurred in the corruption window (the corruption window will lose those — that is the RPO trade-off).
  5. Resume workers.

Expected recovery: Hours, dominated by the PITR restore and the reconciliation. This procedure has not been rehearsed, and its step-by-step runbook will be written at the first restore drill (§ 9). (Roadmap.)

6.5 Region-wide Fly.io outage

Scope: Fly.io's iad region is down for an extended period. Procedure:

  1. Declare a Critical incident per the Incident Response Policy.
  2. Monitor Fly's status page; the first action is to wait, not to rebuild — most regional outages resolve in hours.
  3. If the outage exceeds the threshold at which Fly indicates sustained unrecoverability, rebuild the application in an alternate Fly region using the committed fly.toml configs. This requires:
    • Provisioning a new Managed Postgres cluster in the new region and restoring from the latest available backup.
    • Re-creating Fly secrets in the new apps from 1Password vaults.
    • Updating DNS to point to the new region.
  4. Customer notification follows the incident-response timeline.

Expected recovery: Hours for the first option; a day or more for a full region rebuild. This procedure has not been rehearsed. (Roadmap: regional rebuild drill.)

6.6 Cloudflare R2 outage

Scope: R2 is unavailable; uploads and downloads of customer documents fail. Procedure: Uploads, previews and downloads fail with an error until R2 is back; nothing is queued for later, and the apps show no dedicated "temporarily unavailable" state yet (Roadmap). The database remains writable, so non-document operations (intake, profile changes, return progress) continue. Wait for R2 recovery; no manual restore is meaningful. Expected recovery: Per Cloudflare's published terms.

6.7 GCP KMS outage

Scope: KMS unwrap calls fail. Procedure: Document decryption fails, so previews and downloads answer an error and new uploads wait unsealed until the key service is back; the apps show the failure as an error, not a dedicated notice (Roadmap). The database remains accessible for non-document operations. Wait for GCP recovery. Expected recovery: Per Google Cloud KMS SLA.

6.8 Clerk outage

Scope: Customers cannot sign in; existing sessions continue until their tokens expire. Procedure: Engage Clerk support; surface degraded sign-in to customers. The data remains intact behind authentication; no manual recovery is meaningful. Expected recovery: Per Clerk's published terms.

6.9 Loss of an operator device

Scope: A SendTax operator's laptop is lost, stolen, or compromised. Procedure:

  1. Revoke the operator's GitHub, Fly.io, Vercel, Clerk, GCP, Cloudflare, Resend, Sentry, PostHog, Modal, Google Workspace and 1Password sessions.
  2. Rotate any secrets to which the operator had unique access.
  3. Verify the device's full-disk encryption was active at the time of loss; if not, treat as a potential data-access incident under the Incident Response Policy.
  4. Re-issue access from a known-clean device.

6.10 Key-person unavailability

Scope: One of two operators is unavailable during an incident. Procedure: If the other operator is unavailable, the Qualified Individual, who holds administrative access to our production systems, continues alone. If the Qualified Individual is unavailable, the other operator follows the documented break-glass procedure to deploy a production fix. That procedure covers deploys only; database administration and secret changes wait for the Qualified Individual. Its credentials have not yet been tested, so until they are, a production fix may have to wait for the Qualified Individual. (Roadmap: test the break-glass credentials, and again at an access review at least once a year.) Runbooks for the most common recovery scenarios, including break-glass, are maintained in st-apps/docs/RUNBOOKS.md. For extended unavailability, see the roadmap note in § 5.


7. Roles during recovery

Recovery roles map onto the incident-response roles described in the Incident Response Policy § 5:

  • Incident Commander owns the decision to declare, the severity, and the resolution.
  • Technical Lead owns the recovery procedures in this document.
  • Communications Lead owns customer-facing messaging.

In a two-operator team, one person may hold multiple roles during a small incident. For Critical incidents, the Commander and Communications Lead must be different people whenever both operators are available.


8. Communication during recovery

During recovery, customers are notified through the same audited fan-out infrastructure documented in the Incident Response Policy § 8. Initial notifications go out as soon as we have a defensible factual picture. We do not wait for full recovery before communicating; we send updates as the picture develops.

For events affecting authentication or document availability, we notify all customers. For events affecting only filers or only firms, we scope the notification accordingly. While the apps are up, we can also post a notice inside the filer and preparer apps, which people see the next time they open them.

When a recovery depends on a sub-processor, we follow the Incident Response Policy § 12: we use the provider's support channel, track its updates on our incident timeline, and tell customers what we know.


9. Backup integrity and restore verification

  • Backup creation is performed by Fly.io Managed Postgres. We do not run a parallel backup pipeline. An operator may take a one-off database dump for an operational purpose; dumps follow our Data Retention and Deletion Policy § 3.12.
  • Backup encryption at rest is provided by the managed service.
  • Restore verification — confirming that backups actually restore to a working database — has not yet been formally rehearsed by SendTax. (Roadmap: a documented restore drill at least once a year; the first is due by 2026-11-21.) Once a drill is performed and timed, the measured restore time will become an input to the RTO/RPO statement in § 10.

10. RTO and RPO

SendTax does not currently publish formal RTO or RPO targets.

Our reasoning is straightforward: any RTO or RPO we published today would be a guess derived from provider SLAs rather than a measured number from a tested recovery procedure. We would rather publish nothing than publish a target we cannot defend. We do keep internal planning targets, so the first restore drill has something to be judged against; they stay internal until a drill has measured them.

What we can describe candidly:

  • Effective RPO: bounded by the Managed Postgres PITR window; for the document store, by R2's published durability.
  • Effective RTO: depends entirely on the scenario. Single- machine failures recover in seconds; a region rebuild has not been rehearsed and we will not state a number until it has.

(Roadmap: publish measured RTO and RPO targets after the first restore drill referenced in § 9.)


11. Testing

The components of this plan are exercised in the following ways:

  • Production health checks continuously verify single-machine resilience.
  • Routine deployments of the API, the workers and the filer and preparer apps, to staging (daily, when the code has changed) and to production, pass the same health-check gate as § 6.2. Rolling back to a previous image (§ 6.2) has not been rehearsed.
  • Tabletop exercises are run as the Incident Response Policy § 13 describes; the regional-outage and corruption scenarios (§ 6.4, § 6.5) are among those they can use. None has been run yet.
  • Formal restore drill: at least once a year. None has run yet; the first is due by 2026-11-21. (Roadmap.)

We will publish the dates and results of completed drills in future revisions of this document.


12. Continuous improvement

This plan is updated when any of the following occurs:

  • A real incident reveals a gap.
  • A sub-processor changes its published availability terms.
  • A new component is introduced that changes the BIA in § 3.
  • A roadmap item ships and graduates from § 11 or § 10 into a measured commitment.

Each update is recorded in the change history (§ 15).


13. Policy review

This plan is reviewed at least annually, after any material change, and whenever a Critical or High-severity incident closes. A review that changes nothing is still recorded in the change history (§ 15).


14. Contact

Incident reporting[email protected]
Service status(publishing status page is on the roadmap; in the interim, customers are notified directly during incidents)
General[email protected]
Operating entityHowell & Gibbs LLC

15. Change history

VersionDateAuthorApproved byChanges
1.02026-05-30Holly GibbsNot recordedFirst published text. Drafted with Claude.
1.12026-09-24Liam HowellNot recorded§ 4.3: documents are encrypted on arrival, not before upload; § 3: what happens when Modal is unavailable. Drafted with Claude.
1.22026-10-07Liam HowellLiam Howell (QI), for Howell & Gibbs LLCOne version number and one change history for this plan's internal and public texts and its Notion page (WISP § 7.4; public WISP § 14); earlier versions were numbered separately (internal 1.0–1.1, public 1.0–1.1, Notion draft 1.0). Section numbers below are the public text's. Review cadence: at least annually, after any material change, and when a Critical or High-severity incident closes (the Notion draft said semi-annual). Restore drill: at least once a year, the first by 2026-11-21; none has run yet (§ 9 and § 11 said semi-annual once started). The internal drill procedure now records each drill as a privileged action and deletes the restored copy when it ends. R2 bucket versioning: the roadmap item to enable it is withdrawn (§ 5). Versioning stays off on purpose, as decided on 2026-09-25 (internal 1.1), because a versioned bucket would keep a copy of each document from before it was encrypted and of every deleted document; a document deleted by mistake can't be restored. Corrected, not changes in practice: everything on Fly.io runs in one region, iad, and only the marketing site is on Vercel (§ 3 and § 4.1 had put it on Fly; the internal text said two regions and all web apps on Vercel); a failing health check does not restart a machine (§ 4.1 and § 6.1); there is no on-call rotation, so an operator investigates restarts that loop (§ 6.1 said the on-call operator); a missed Beat check-in sends an email alert to the operators and pages no one's phone yet (§ 5 said it paged the on-call); the break-glass procedure covers deploys only, and its credentials have not yet been tested (§ 5 said both operators could deploy production fixes, and § 6.10 that the remaining operator had full administrative access to every system); an operator may take a one-off database dump (§ 9 said we produce no parallel backups); routine deploys don't rehearse a rollback (§ 11); provisioning sets Google Cloud KMS sign-in to Workload Identity Federation, but the mode production uses is not yet confirmed (§ 4.4 said production uses it); the internal text's database-replica and 35-day-snapshot statements, which couldn't be verified, are removed. Added: the break-glass kit is checked at an access review at least once a year (§ 6.10); internal RTO/RPO planning targets stay unpublished until a drill measures them (§ 10); the boundary with the Incident Response Policy (§ 1); PITR defined (§ 2); the database-restore runbook is still to be written (§ 6.4); Vercel and Google Workspace in the device-loss steps (§ 6.9); in-app notices and sub-processor coordination (§ 8); tabletop exercises follow the Incident Response Policy (§ 11). Dates corrected to the day each version merged: public 1.0 2026-05-30 (shown as effective 2026-05-21), public 1.1 2026-09-24 (shown as revised 2026-10-01), internal 1.1 2026-09-25 (was 2026-09-24). Also records changes made without a row: #150 (2026-06-08, § 5 key-person row: break-glass deploys) and #155 (2026-06-08, § 4.1 two API machines; § 5 Beat watchdog). Two changes narrow what the plan promised, so they apply only from the day this version is published on send.tax: withdrawing R2 versioning (§ 5), and restore drills once a year rather than twice (§§ 9, 11). No notice is owed: SendTax has no customers yet. Drafted with Claude.
1.32026-10-22Liam HowellLiam Howell (QI), for Howell & Gibbs LLCSecurity statements, the Qualified Individual's decision of 2026-10-21: when SendTax describes itself it doesn't name SOC 2 or its criteria. Internal § 1 no longer lists SOC 2 criteria the plan addresses (CC7.5, A1.2 and A1.3); no outside firm has assessed SendTax against them. The public text is unchanged. Drafted with Claude.
1.42026-10-26Liam HowellLiam Howell (QI), for Howell & Gibbs LLCThe 2026-10-26 security and legal review. Corrected, internal § 3 and public §§ 4.3 and 5: Cloudflare R2 offers no object versioning, so there was nothing to decline; SendTax keeps no second copy of the bucket, which is the one copy of each document, and a document lost to a compromised storage credential can't be restored either (the text said versioning was deliberately not enabled and gave geographic distribution as a resilience property). A copy of encrypted objects only, in a second account under a short lock, is under consideration (Roadmap). Public §§ 6.6 and 6.7 corrected: during an R2 or KMS outage the apps show errors; nothing is queued and there is no dedicated "temporarily unavailable" state (the text said the UI surfaces it and queues operations). Internal § 3: the 2026-07-22 dumps' R2 copies went with the old production bucket on 2026-10-22; the laptop copy remains, unrecorded, and deleting it is an open item. Drafted with Claude.