Skip to content
CivicDogCivicDog

Infrastructure

Every AWS resource CivicDog runs on is provisioned by Terraform, in a dedicated cd-infra repo, kept completely separate from application code.

Terraform layout

Section titled “Terraform layout”
cd-infra/terraform/
├── bootstrap/    # State bucket (S3 + KMS) + GitHub OIDC provider — one-time, local state
├── networking/   # VPC, subnets, security groups
├── rds/          # Single-AZ Postgres
├── airflow-ecs/  # ECS cluster (EC2) running cd-etl as 4 Airflow services
├── cd-api/       # Lambda + API Gateway + RDS Proxy + api.civicdog.com + OpenAPI bucket
├── cd-server/    # ECS cluster (EC2) + ALB for the GraphQL app backend
├── cd-webapp/    # Cognito user pool + SES email + Amplify app for app.civicdog.com
└── amplify/      # Amplify apps for civicdog.com and docs.civicdog.com

Each directory is an independent root module with its own state — there’s no single terraform apply for everything. Later modules read earlier ones’ outputs via terraform_remote_state. State lives in S3 with Terraform’s native use_lockfile locking (Terraform ≥ 1.10) — no DynamoDB lock table needed.

terraform apply is manual-only, by design — there’s no auto-apply workflow. Every PR runs terraform fmt -check, terraform validate, and a Trivy IaC security scan; a human runs apply.

What’s provisioned

Section titled “What’s provisioned”
Category Services
Compute ECS on EC2: Airflow (4 services) and cd-server, on separate clusters; Lambda (cd-api)
Load balancing & API ALB for server.civicdog.com; API Gateway REST API with API keys + usage plan, on an edge-optimized api.civicdog.com custom domain
Database RDS Postgres (single-AZ, KMS-encrypted), RDS Proxy
Auth & email Cognito user pool with managed login on a custom domain; SES for its email, with DKIM/SPF/DMARC
Hosting Amplify for the React app, the landing page, and these docs
AI Bedrock (Titan Text Embeddings V2 for Semantic Search; a chat model for AI voting-record summaries)
Networking VPC (2 AZs), public/private subnets, single NAT Gateway, security groups; DNS on Cloudflare
IAM Scoped roles per service and per ECS task, plus one GitHub OIDC deploy role per service
Secrets Secrets Manager (Congress API key, per-service DB credentials, Airflow admin/JWT secrets), 5 customer-managed KMS keys
Storage S3 (Terraform state, versioned and encrypted; a public bucket serving the OpenAPI spec for these docs)

Both ECS clusters use the EC2 launch type, not Fargate: cheaper for always-on workloads at this scale. There’s still no EKS and no VPN.

Why Airflow moved to ECS

Section titled “Why Airflow moved to ECS”

Airflow originally ran as a single airflow standalone container on a plain EC2 instance. Then a subprocess inside it died silently: the container still looked healthy while the pipeline missed five days of runs. A restart policy only reacts when the whole container exits, so it never fired. Splitting Airflow into four ECS services (scheduler, DAG processor, triggerer, API server), each with its own health check, means ECS replaces whichever process actually stops working. The new cluster ran alongside the old instance until it was proven, then the original was decommissioned and its KMS key and secrets were moved over with terraform state mv rather than recreated.

Security posture

Section titled “Security posture”
  • Zero public IPs on compute. RDS, both ECS clusters’ instances, and Lambda all live in private subnets. The only public entry points are managed: API Gateway, the ALB, and Amplify.
  • SSM Session Manager, not a bastion or VPN. The only way to shell into an ECS host or port-forward to RDS is through SSM — no open SSH port, no VPN infrastructure to maintain.
  • Five separate customer-managed KMS keys (state bucket, RDS, Airflow, cd-api, cd-server), each with rotation enabled — a resource compromise in one domain doesn’t grant decrypt access to another.
  • No static DB credentials at runtime. The RDS master password is Secrets-Manager-managed by RDS itself (never touches Terraform state as plaintext). App-level roles (cd_etl_app, cd_api_app, cd_server_app) are least-privilege, generated by Terraform, and bootstrapped into Postgres out-of-band.
  • IAM built empirically. The Terraform deployer’s own IAM policy was built up one real AccessDenied error at a time in the console, rather than guessed from docs upfront — a deliberate practice, not an oversight.

Cost trade-offs, quantified

Section titled “Cost trade-offs, quantified”

Real numbers, documented in the repo rather than assumed:

Decision Alternative considered Cost
Self-hosted Airflow on ECS (one EC2 t3.medium) AWS MWAA Under ~$40/mo (actual billed EC2) vs. ~$358/mo
ECS on EC2 ECS on Fargate Fixed instance cost for always-on services, no per-task premium
SSM Session Manager AWS Client VPN Free vs. ~$72/mo per subnet association
Single shared NAT Gateway One NAT Gateway per AZ ~$32/mo vs. ~$64/mo

Known gaps

Section titled “Known gaps”

Documented honestly rather than hidden:

  • No CloudWatch alarms or dashboards yet. Lambda and API Gateway get default CloudWatch Logs, but there’s no alerting on top of them.
  • Single environment. Everything runs in one AWS account with no dev/staging split — API Gateway’s stage is literally named prod. This is a deliberate MVP trade-off (“flip it on in whichever PR first needs it, not preemptively”), not an oversight.
  • API key auth is a stopgap. cd-api is currently protected by a static API Gateway key rather than something more granular like IAM auth or OAuth scopes.