Business Continuity & Disaster Recovery

This document describes how SendTax keeps the service available during disruption, what we do when a disruption becomes a disaster, and how we recover. It is a companion to our Information Security, Incident Response, and Vendor Management policies.

We are publishing this plan as a candid description of our current posture at the company's current stage. We are not claiming multi-region active-active deployment, validated recovery-time objectives, or capabilities we have not built. Items marked Roadmap are deliberate next steps.

---

1. Scope

This plan covers:

• The technical resilience of the SendTax service.
• The procedures we follow when a component fails or a region becomes unavailable.
• Our preparations for events that affect the SendTax operating team itself (key-person unavailability, loss of operator device).

It does not cover financial business continuity for Howell & Gibbs LLC as a company; that is addressed separately.

---

2. Definitions

---

3. Business Impact Analysis

In recovery, components are restored in this priority order:

---

4. Resilience controls currently in place

This section describes what is actually deployed today.

4.1 Application layer (Fly.io)

• Primary region: All SendTax application machines (API, workers, web frontends) run in Fly.io's iad region (US-East, Ashburn, Virginia). This is configured explicitly in every fly.toml in the repo.
• Per-machine resilience: Fly automatically restarts machines on health-check failure. The API service is configured with min_machines_running = 2 and auto_start_machines = true — so a single API machine failure leaves a live peer serving traffic; workers have [[restart]] policy = "always" with up to 5 retries.
• Health checks: /health/ on the API and a broker-connectivity sidecar on the worker, both polled every 30 seconds by Fly.
• Auto-rollback: Failed health checks block deployments and surface in fly checks list.

4.2 Database layer (Fly.io Managed Postgres)

• Managed service: Production uses Fly.io Managed Postgres (st-pg-prod), which provides automatic failover and managed high availability per Fly's published terms.
• Backups: Automated daily backups with point-in-time recovery within the managed plan's retention window. We do not maintain a separate backup pipeline.
• Encryption: Storage and backups are encrypted at rest by Fly's managed service.

4.3 Document storage (Cloudflare R2)

• Durability: R2 provides Cloudflare's published durability guarantees and geographic distribution across the Cloudflare network.
• Encryption layers: Documents are AES-256-GCM-encrypted by the SendTax application before upload; R2 then applies its own at-rest encryption — so customer documents are encrypted twice, with SendTax holding the key-management end of the outer layer (see Encryption Policy).

4.4 Key management (Google Cloud KMS)

• Managed service: KEK custody is in GCP Cloud KMS, region us-east1. KMS is operated by Google and inherits its SLA.
• Authentication: Workload Identity Federation; no long-lived service-account keys on application servers.
• Dual-key migration path: The encryption module supports pointing wrap operations at a new KEK while still being able to unwrap DEKs that were wrapped under a legacy KEK — usable as a rolling re-encryption path in a key compromise.

4.5 Identity (Clerk)

• Managed service: Authentication is delegated to Clerk, which operates its own multi-region redundancy. We inherit Clerk's availability for sign-in flows.

4.6 Operational resilience

• Infrastructure as code: Fly configurations are committed to the monorepo (fly.api.prod.toml, fly.worker.prod.toml, and the per-app fly.toml files). The cluster can be re-provisioned from version control.
• Secrets recovery: Application secrets are stored both in Fly encrypted secrets and (operator-accessible) in environment-scoped 1Password vaults, so the loss of either does not prevent rebuilding the cluster.

---

5. Single points of failure (and how we handle them)

We deliberately enumerate these rather than hide them.

---

6. Threat scenarios and recovery procedures

We organize procedures by failure scope. Each procedure is the intended action; the Incident Response Policy governs the broader response (declaration, communication, postmortem).

6.1 Single machine failure

Scope: One Fly machine fails (process crash, VM death). Detection: Fly health checks fail; Sentry may surface symptoms. Procedure: Fly auto-restarts the machine. No operator action is required in most cases. If auto-restart loops, the on-call operator investigates and may roll back the most recent deployment. Expected recovery: Seconds to minutes.

6.2 Application deployment failure

Scope: A bad deploy makes the API or worker unhealthy. Detection: Failed health checks block deployment; Sentry surfaces errors post-deploy if a subtle regression slips through. Procedure: Roll back to the previous image tag (fly deploy --image <previous-tag>). Investigate after rollback; do not debug in production. Expected recovery: Minutes.

6.3 Database outage (within Fly.io managed service)

Scope: Managed Postgres becomes unavailable but the cluster is still under Fly's control. Detection: API health checks fail with database errors; Sentry surfaces connection failures. Procedure: Engage Fly.io support immediately. Fly's managed service is responsible for failover; we monitor and communicate. Expected recovery: Per Fly Managed Postgres SLA.

6.4 Data corruption requiring restore

Scope: Logical corruption (a bad migration, application bug writing wrong data) where the live database is intact but contains incorrect data. Procedure:

1. Stop write-path workers immediately (fly scale count 0 on st-worker-prod).
2. Determine the corruption window (when did the bad writes begin?).
3. Engage Fly support to restore from PITR to the latest pre-corruption point.
4. Reconcile any legitimate writes that occurred in the corruption window (the corruption window will lose those — that is the RPO trade-off).
5. Resume workers.

Expected recovery: Hours, dominated by the PITR restore and the reconciliation.

6.5 Region-wide Fly.io outage

Scope: Fly.io's iad region is down for an extended period. Procedure:

1. Declare a Critical incident per the Incident Response Policy.
2. Monitor Fly's status page; the first action is to wait, not to rebuild — most regional outages resolve in hours.
3. If the outage exceeds the threshold at which Fly indicates sustained unrecoverability, rebuild the application in an alternate Fly region using the committed fly.toml configs. This requires:
   • Provisioning a new Managed Postgres cluster in the new region and restoring from the latest available backup.
   • Re-creating Fly secrets in the new apps from 1Password vaults.
   • Updating DNS to point to the new region.
4. Customer notification follows the incident-response timeline.

Expected recovery: Hours for the first option; a day or more for a full region rebuild. This procedure has not been rehearsed. (Roadmap: regional rebuild drill.)

6.6 Cloudflare R2 outage

Scope: R2 is unavailable; uploads and downloads of customer documents fail. Procedure: Customer document operations degrade gracefully — the UI surfaces "temporarily unavailable" messages and queues operations where safe. The database remains writable, so non- document operations (intake, profile changes, return progress) continue. Wait for R2 recovery; no manual restore is meaningful. Expected recovery: Per Cloudflare's published terms.

6.7 GCP KMS outage

Scope: KMS unwrap calls fail. Procedure: Document decryption fails. The UI surfaces the condition. The database remains accessible for non-document operations. Wait for GCP recovery. Expected recovery: Per Google Cloud KMS SLA.

6.8 Clerk outage

Scope: Customers cannot sign in; existing sessions continue until their tokens expire. Procedure: Engage Clerk support; surface degraded sign-in to customers. The data remains intact behind authentication; no manual recovery is meaningful. Expected recovery: Per Clerk's published terms.

6.9 Loss of an operator device

Scope: A SendTax operator's laptop is lost, stolen, or compromised. Procedure:

1. Revoke the operator's GitHub, Fly.io, Clerk, GCP, Cloudflare, Resend, Sentry, PostHog, Modal, and 1Password sessions.
2. Rotate any secrets to which the operator had unique access.
3. Verify the device's full-disk encryption was active at the time of loss; if not, treat as a potential data-access incident under the Incident Response Policy.
4. Re-issue access from a known-clean device.

6.10 Key-person unavailability

Scope: One of two operators is unavailable during an incident. Procedure: The remaining operator has full administrative access to every system. Runbooks for the most common recovery scenarios are maintained in st-apps/docs/RUNBOOKS.md. For extended unavailability, see the roadmap note in § 5.

---

7. Roles during recovery

Recovery roles map onto the incident-response roles described in the Incident Response Policy § 5:

• Incident Commander owns the decision to declare, the severity, and the resolution.
• Technical Lead owns the recovery procedures in this document.
• Communications Lead owns customer-facing messaging.

In a two-operator team, one person may hold multiple roles during a small incident. For Critical incidents, the Commander and Communications Lead must be different people whenever both operators are available.

---

8. Communication during recovery

During recovery, customers are notified through the same audited fan-out infrastructure documented in the Incident Response Policy § 8. Initial notifications go out as soon as we have a defensible factual picture. We do not wait for full recovery before communicating; we send updates as the picture develops.

For events affecting authentication or document availability, we notify all customers. For events affecting only filers or only firms, we scope the notification accordingly.

---

9. Backup integrity and restore verification

• Backup creation is performed by Fly.io Managed Postgres. We do not produce parallel backups.
• Backup encryption at rest is provided by the managed service.
• Restore verification — confirming that backups actually restore to a working database — has not yet been formally rehearsed by SendTax. (Roadmap: documented restore drill on a semi-annual cadence.) Once a drill is performed and timed, the measured restore time will become an input to the RTO/RPO statement in § 10.

---

10. RTO and RPO

SendTax does not currently publish formal RTO or RPO targets.

Our reasoning is straightforward: any RTO or RPO we published today would be a guess derived from provider SLAs rather than a measured number from a tested recovery procedure. We would rather publish nothing than publish a target we cannot defend.

What we can describe candidly:

• Effective RPO: bounded by the Managed Postgres PITR window; for the document store, by R2's published durability.
• Effective RTO: depends entirely on the scenario. Single- machine failures recover in seconds; a region rebuild has not been rehearsed and we will not state a number until it has.

(Roadmap: publish measured RTO and RPO targets after the semi-annual restore drill referenced in § 9.)

---

11. Testing

The components of this plan are exercised in the following ways:

• Production health checks continuously verify single-machine resilience.
• Routine deployments rehearse the rollback procedure roughly weekly.
• Tabletop exercises for regional-outage and corruption scenarios are on our roadmap and have not yet been performed. (Roadmap.)
• Formal restore drill is on our roadmap on a semi-annual cadence once initiated. (Roadmap.)

We will publish the dates and results of completed drills in future revisions of this document.

---

12. Continuous improvement

This plan is updated when any of the following occurs:

• A real incident reveals a gap.
• A sub-processor changes its published availability terms.
• A new component is introduced that changes the BIA in § 3.
• A roadmap item ships and graduates from § 11 or § 10 into a measured commitment.

Each update is recorded in the version line at the top of this document.

---

13. Policy review

This plan is reviewed at least annually and additionally whenever a Critical or High-severity incident closes.

---

14. Contact