Production (GCP)

This is the TARGET production topology, not today's serving environment

Olly's live, traffic-serving environment today is Docker Compose on two Hetzner dev VMs - dev-1 (10.0.1.1, Python AI services: triage, balance, content, nhs111) and dev-2 (10.0.1.2, one Postgres process, the Go services, APISIX, Keycloak, Temporal, OpenBao, Kafka, OpenSearch, and the OTel/Prometheus/Grafana/Loki/Tempo stack). There is no cloud production cutover yet: no DNS points at GCP, no application pods serve traffic from GKE, and APISIX/OpenBao/nginx are still the live gateway/secrets/TLS.

This page describes the GCP migration target (project olly-prod, region europe-west2), an active but incomplete migration tracked in /gcp/MIGRATION_STATUS.md. The Status column below reflects what has been provisioned in that migration repo (/gcp/terraform, 13 OpenTofu roots), not what serves production traffic. A row marked live means the GCP resource exists and is validated (e.g. Cloud SQL holds a restored copy of the 22 databases) - it does not mean the platform runs on it. Until the sub-project #4/#5 app-deploy + DNS cutover completes, treat the running system as the Docker Compose stack described in local-dev.md.

Environment scope: This document describes the production GCP target (project olly-prod, region europe-west2). The live dev stack (Docker Compose on Hetzner VMs) is described in local-dev.md. Provisioned vs pending status for every GCP resource is in /gcp/MIGRATION_STATUS.md.

What runs where

Resource	GCP service	Spec / name	Status
Compute	GKE Standard private cluster	`olly-prod-ew2`, europe-west2, autoscaling	live
Relational DB	Cloud SQL	`prod-pg-16` - PG16 db-custom-2-7680, regional HA, CMEK, IAM auth @ `10.10.49.2`	live - 22 DBs migrated
Cache	Memorystore Valkey	`prod-valkey` - BASIC 1 GB, TLS+AUTH @ `10.10.48.3:6378`	live
Message broker	Strimzi 1.0.0 + Kafka 4.0 KRaft	`data-plane/olly-kafka` - 3-broker `controller` KafkaNodePool, 30 Gi PD-SSD each	live
Search	OpenSearch helm `2.37.0`	`data-plane/opensearch-cluster-master-0`, 20 GB PVC	live (indexes regenerated from DB)
Object storage	GCS + Cloud CDN	Per-service buckets, CMEK key `gcs-prod`	provisioned
Secrets	GCP Secret Manager	`olly-prod` project, `europe-west2` replication	live - replacing OpenBao
API gateway	GCLB + GKE Gateway API	`prod-ext-gw`, `gke-l7-global-external-managed`, Cloud Armor	pending (sub-project #5)
TLS	Google-managed certs via Certificate Manager	Certmap `dev-k8s-hiolly-com-certmap`	pending (sub-project #5)
Frontend hosting	GCS + Cloud CDN behind GCLB	Per-app buckets	pending (sub-project #5)
DNS	Cloud DNS zone `hiolly-com`	`olly-shared` project; NS records not yet pointed	pending (sub-project #5)
Observability (interim)	Cloud Logging + Managed Prometheus	Enabled on GKE cluster	live
Observability (full)	Grafana + Tempo + Loki on GKE	Self-hosted; sub-project #6	pending
Notifications	Novu on GKE	`data-plane/novu-*` (MongoDB + Redis bundled)	pending (sub-project #4)
Audit	BigQuery sink	`audit_logs` dataset in `olly-mgmt`	live
Encryption keys	Cloud KMS keyring `olly-prod`	`cloud-sql-prod`, `gke-etcd-prod`, `gcs-prod` (90-day rotation)	live

GCP projects

Project	Purpose
`olly-prod`	All production workloads: GKE, Cloud SQL, Memorystore, GCS, Secret Manager
`olly-shared`	Artifact Registry (5 repos), Cloud DNS zone `hiolly-com`, shared KMS keyring
`olly-mgmt`	Terraform state bucket `olly-mgmt-tf-state`, audit log BigQuery sink, billing
`olly-platform-dev`	Dev-parity fallback (not yet built out)

Org: ollyinsurance.com (ID 297857843503). Folders: Platform/, Sandbox/.

Kubernetes namespace structure

olly-prod-ew2/
├── gateway/       # GKE Gateway API resources (Gateway, HTTPRoute CRs)
├── data-plane/    # Stateful workloads: Kafka (Strimzi), OpenSearch, Keycloak,
│                  # Temporal, Mirth, Langfuse, Novu, GrowthBook, novu-mongo,
│                  # growthbook-mongo
├── apps/          # All Go microservices (claims, billing, enrollment, …)
└── default/       # cloud-sql-proxy sidecar SA; miscellaneous

Node pools:

stateful-pool - e2-standard-4, pd-ssd 100 GB, taint workload-class=stateful:NoSchedule, 3-6 nodes (1-2 per zone). Stateful workloads tolerate this taint.
general-pool - added in sub-project #4 for application pods; not yet provisioned.

Provisioning: Terraform roots

All IaC lives in /gcp/terraform/. Apply roots in numeric order; each is an independent OpenTofu root with its own state file in gs://olly-mgmt-tf-state/.

Root	Provisions
`00-org-policies/`	10 org policies (shielded VMs, uniform bucket access, restrict public IPs, …)
`10-iam-groups/`	5 Cloud Identity groups + project-level IAM bindings
`20-shared-resources/`	Artifact Registry repos, Cloud DNS `hiolly-com` zone, shared KMS keyring
`30-prod-network/`	prod-vpc, prod-vpc-ew2 subnet, secondary ranges (pods/services), Cloud NAT `prod-nat`
`40-dev-network/`	dev-vpc (reserved; not yet used)
`50-data-plane-foundation/`	KMS keyring `olly-prod` + 3 keys, PSA peering ranges for Cloud SQL + Memorystore
`60-cloud-sql/`	Cloud SQL `prod-pg-16`, 22 databases, IAM auth, SA `wi-cloudsql-proxy`
`65-memorystore/`	Memorystore Valkey `prod-valkey`
`70-gke-cluster/`	GKE cluster `olly-prod-ew2`, `stateful-pool`, 6 Workload Identity SAs + bindings
`75-vpn-hetzner/`	Reserved; DEF-005 closed - Cloud VPN never needed (Kafka is on GKE)
`80-gcs-backup-buckets/`	6 backup GCS buckets (CMEK, versioning, lifecycle)
`85-data-plane-secrets/`	7 random-password secrets in Secret Manager for stateful helm charts
`90-mgmt-resources/`	BigQuery audit sink, Terraform state bucket

bash

# Example: apply the GKE cluster root
cd /gcp/terraform/70-gke-cluster
tofu init && tofu plan && tofu apply

Workload Identity

Workload Identity replaces IRSA. The pattern: one GCP Service Account (GSA) per workload, annotated K8s ServiceAccount (KSA) bound to it.

Defined in /gcp/terraform/70-gke-cluster/main.tf (for stateful workloads) and /gcp/terraform/60-cloud-sql/main.tf (for the proxy SA).

Existing SA bindings

GSA (`olly-prod`)	K8s namespace/SA	IAM roles
`wi-langfuse@`	`data-plane/langfuse`	`secretmanager.secretAccessor`
`wi-opensearch@`	`data-plane/opensearch`	(GCS read for snapshots - add when needed)
`wi-novu-mongo@`	`data-plane/novu-mongo`	`secretmanager.secretAccessor`
`wi-growthbook-mongo@`	`data-plane/growthbook-mongo`	`secretmanager.secretAccessor`
`wi-valkey-client@`	`default/valkey-client`	`redis.viewer`
`wi-cloudsql-proxy@`	`default/cloudsql-proxy`	`cloudsql.client`, `cloudsql.instanceUser`

Binding a new workload

Add a GSA + google_service_account_iam_member in the relevant Terraform root:

hcl

resource "google_service_account" "wi_my_service" {
  project      = var.project_prod
  account_id   = "wi-my-service"
  display_name = "Workload Identity SA for my-service"
}

resource "google_service_account_iam_member" "wi_my_service_binding" {
  service_account_id = google_service_account.wi_my_service.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "serviceAccount:${var.project_prod}.svc.id.goog[apps/my-service]"
  depends_on         = [google_container_cluster.prod]
}

Annotate the K8s ServiceAccount (in your Helm values or manifest):

yaml

serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: wi-my-service@olly-prod.iam.gserviceaccount.com

Run tofu apply in the relevant root, then kubectl rollout restart deployment/my-service -n apps.

Secret Manager: access pattern

Secrets are stored in olly-prod, replicated to europe-west2 only. The pod must run as a KSA bound (via Workload Identity) to a GSA that has roles/secretmanager.secretAccessor on the secret or project.

Reading a secret from inside a pod

Option A - CSI driver (preferred; mounts secret as a file):

yaml

# In the Pod spec:
volumes:
  - name: my-secret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: my-service-secrets
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-service-secrets
  namespace: apps
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: projects/olly-prod/secrets/my-api-key/versions/latest
        path: my-api-key

Option B - gcloud SDK / REST from application code (acceptable for bootstrap scripts):

bash

gcloud secrets versions access latest \
  --secret=my-api-key \
  --project=olly-prod

Option C - Secret Manager Go SDK (for Go services):

client, _ := secretmanager.NewClient(ctx) // uses ADC / Workload Identity
req := &secretmanagerpb.AccessSecretVersionRequest{
    Name: "projects/olly-prod/secrets/my-api-key/versions/latest",
}
result, _ := client.AccessSecretVersion(ctx, req)
payload := result.Payload.Data

DEF-009: External Secrets Operator (ESO) is the long-term target for syncing Secret Manager secrets into K8s Secrets. Currently bypassed - pods read secrets via CSI driver or SDK directly.

Kafka: Strimzi KafkaTopic CRs

Kafka runs as Strimzi-managed KRaft (no ZooKeeper). Topics are managed via KafkaTopic custom resources; do not create topics via kafka-topics.sh.

Manifest: /gcp/k8s/data-plane/kafka/kafka-cluster.yaml

Publishing a new topic

yaml

apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
  name: my-new-topic          # K8s resource name (must be unique in namespace)
  namespace: data-plane
  labels:
    strimzi.io/cluster: olly-kafka
spec:
  partitions: 3
  replicas: 3
  topicName: my.new.topic     # Actual Kafka topic name (omit if same as metadata.name)
  config:
    retention.ms: "604800000" # 7 days; omit to use broker default

bash

kubectl apply -f my-new-topic.yaml
kubectl get kafkatopic -n data-plane   # verify reconciled

Existing topics

KafkaTopic CR	Kafka topic name	Partitions	Replicas
`billing`	`billing`	3	3
`billing-events`	`billing.events`	3	3
`care-events`	`care.events`	3	3
`claims-events`	`claims.events`	3	3
`consent-events`	`consent.events`	3	3
`enrollment`	`enrollment`	3	3
`enrollment-events`	`enrollment-events`	3	3
`enrollment-events-dot`	`enrollment.events`	3	3

Cluster config: min.insync.replicas=2, default.replication.factor=3, offsets.topic.replication.factor=3.

Edge, ingress, and TLS (sub-project #5 - pending)

Gateway API

APISIX is retired; replaced by GKE Gateway API backed by GCLB.

The prod-ext-gw Gateway is defined in /gcp/k8s/gateway/gateway.yaml. It uses gatewayClassName gke-l7-global-external-managed, binds to static IP prod-gclb-ip, and serves *.dev.k8s.hiolly.com via Certificate Manager certmap dev-k8s-hiolly-com-certmap.

Service-per-subdomain routes

Each Go service gets its own HTTPRoute:

yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: claims-route
  namespace: apps
spec:
  parentRefs:
    - name: prod-ext-gw
      namespace: gateway
  hostnames:
    - "claims.dev.k8s.hiolly.com"
  rules:
    - backendRefs:
        - name: claims
          port: 8080

All routes are in /gcp/k8s/gateway/gateway.yaml.

API gateway routes (replaces APISIX path-based routing)

api.dev.k8s.hiolly.com routes are path-prefix rules in /gcp/k8s/gateway/api-routes.yaml. Example:

yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-claims
  namespace: apps
spec:
  parentRefs:
    - name: prod-ext-gw
      namespace: gateway
  hostnames: ["api.dev.k8s.hiolly.com"]
  rules:
    - matches:
        - path: {type: PathPrefix, value: /claims}
      filters:
        - type: URLRewrite
          urlRewrite:
            path: {type: ReplacePrefixMatch, replacePrefixMatch: /}
        - type: RequestHeaderModifier
          requestHeaderModifier:
            set:
              - {name: X-Market, value: GB}
      backendRefs:
        - name: claims
          port: 8080

Consumer-facing routes (claims, billing, eligibility, enrollment, notifications, provider) inject X-Market: GB.

Google-managed certs

TLS is terminated at GCLB. The Certificate Manager certmap annotation on the Gateway (networking.gke.io/certmap: dev-k8s-hiolly-com-certmap) binds a wildcard Google-managed cert. No cert-manager or Let's Encrypt is used in prod.

Observability

Signal	Source	Current sink	Future sink (sub-project #6)
Logs	GKE workload stdout/stderr	Cloud Logging (GKE integration)	Cloud Logging + Loki on GKE
Metrics	GKE system + `ServiceMonitor` CRs	Managed Prometheus (GKE)	Managed Prometheus + Grafana
Traces	OTel SDK (`github.com/olly/middleware`)	Cloud Trace (interim)	Tempo on GKE
LLM traces	Langfuse web + worker on GKE	Langfuse ClickHouse (bundled)	Same; 69 GB history deferred (DEF-008)
Audit	GCP audit logs	BigQuery `audit_logs` in `olly-mgmt`	Same (permanent)

Every Go service is instrumented at startup via github.com/olly/middleware. Traces, metrics, and logs are correlated by trace_id and service.name.

Notifications

All notifications (email, SMS, push, in-app) flow through Novu. Services must not call SendGrid / Twilio / FCM / SMTP directly.

Novu components deploy to GKE data-plane/ namespace in sub-project #4 (Novu MongoDB pod scheduling in progress). Public endpoints (once sub-project #5 DNS cutover completes):

URL	Purpose
`https://notifications.hiolly.com`	Novu dashboard - workflow authoring + delivery logs
`https://notifications-api.hiolly.com`	REST API - `POST /v1/events/trigger`
`https://notifications-ws.hiolly.com`	WebSocket - in-app inbox + bell badge

Compliance

Data at rest: CMEK via Cloud KMS (cloud-sql-prod, gke-etcd-prod, gcs-prod). 90-day key rotation.
Data in transit: TLS 1.3 enforced at GCLB; inter-pod traffic policy enforced via Calico network policies.
Audit: all Admin Read / Data Write / Data Read GCP API calls flow to BigQuery audit_logs in olly-mgmt. Retention governed by BigQuery table expiry policy (configure before cutover).
PHI scope: Cloud SQL IAM auth only; no long-lived DB passwords in env vars. Service accounts use Workload Identity; no SA key files.
OPA enforces fine-grained RBAC/ABAC on every API request (deployed alongside app pods in sub-project #4).

Migration status

See /gcp/MIGRATION_STATUS.md for the live checklist.

Current sub-project: #3 prod data plane (Cloud SQL + Memorystore + GKE + Strimzi + 4 stateful workloads) - largely complete. Next: #5 Edge & ingress (DNS cutover + managed certs + GCLB URL maps). Pending app deploys: Go microservices, Triage, NHS111, Keycloak, Temporal, Mirth (sub-project #4).

Retired (not migrated): APISIX (→ GCLB), OpenBao (→ Secret Manager), Let's Encrypt + nginx (→ Google-managed certs + GCLB), Alloy shipper, Jaeger, Gatus, uat-clone soak services.

Production (GCP) ​

What runs where ​

GCP projects ​

Kubernetes namespace structure ​

Provisioning: Terraform roots ​

Workload Identity ​

Existing SA bindings ​

Binding a new workload ​

Secret Manager: access pattern ​

Reading a secret from inside a pod ​

Kafka: Strimzi KafkaTopic CRs ​

Publishing a new topic ​

Existing topics ​

Edge, ingress, and TLS (sub-project #5 - pending) ​

Gateway API ​

Service-per-subdomain routes ​

API gateway routes (replaces APISIX path-based routing) ​

Google-managed certs ​

Observability ​

Notifications ​

Compliance ​

Migration status ​

Production (GCP)

What runs where

GCP projects

Kubernetes namespace structure

Provisioning: Terraform roots

Workload Identity

Existing SA bindings

Binding a new workload

Secret Manager: access pattern

Reading a secret from inside a pod

Kafka: Strimzi KafkaTopic CRs

Publishing a new topic

Existing topics

Edge, ingress, and TLS (sub-project #5 - pending)

Gateway API

Service-per-subdomain routes

API gateway routes (replaces APISIX path-based routing)

Google-managed certs

Observability

Notifications

Compliance

Migration status