Change Control & Deployment

Position

Olly ships code via a trunk-based workflow: every change lands on main through a single-approver PR, every merge is automatically promoted dev → uat → prod, and the moment a feature is enabled for users is deliberately separated from the moment its code is deployed via GrowthBook feature flags. The explicit boundary: nothing reaches production without passing CI, and no runtime behaviour changes outside a feature flag, a migration, or a redeploy - there is no out-of-band edit path.

Branch strategy (trunk-based; short-lived feature branches; squash-merge)

main is the only long-lived branch. Engineers cut a short-lived feature branch (hours to a few days, never weeks) and squash-merge back, giving main a linear, one-commit-per-change history that is trivial to bisect, revert, and read in release notes.

• Rebase, don't merge: feature branches rebase on main; no merge commits land on main.
• No release branches and no long-running integration branches: integration happens on main, gated by flags for work not yet ready to be live.

Code review (single approver; tests must be green; security scan must be clean)

Every PR requires one approving reviewer other than the author, with all CI checks green, before it can merge. The single-approver model keeps review latency low without trading away the second-pair-of-eyes property that catches the bugs static checks miss.

• Review covers correctness, security posture, observability (spans + logs present), and the coding guidelines.
• A structured checklist (2PC outbox, OTel spans, error handling, migration safety) is run on every PR.
• Red CI, an unresolved security finding, or an unresolved reviewer comment all block merge.

INFO

Self-approval is not permitted, even for one-line fixes.

CI pipeline (lint + unit + integration + security scan + SBOM)

CI runs on every PR and every push to main. Each stage is a hard gate; failure blocks the merge or the promotion.

1. Lint & format - language-native linters plus repo-wide guardrails (no committed secrets, no replace directives, no leftover TODO).
2. Unit tests - pure logic, no I/O; sub-minute runtime so engineers run them locally.
3. Integration tests - service-level tests against real Postgres + Kafka via testcontainers. No DB mocking; we test against the engines we ship.
4. End-to-end tests - cross-service flows exercising the 2PC outbox, Kafka event propagation, and OTel trace correlation.
5. Security scan - SAST, dependency, container-image, and IaC scans. Findings at "high" or above block the merge.
6. SBOM - every built image emits a CycloneDX SBOM, attached to the artifact and retained for its lifetime as the supply-chain evidence trail.

Runners live inside the Olly VPC, so build outputs and SBOMs stay inside the perimeter.

CD pipeline (dev -> uat -> prod; manual approval at the uat -> prod boundary)

Merges to main auto-deploy to dev. After a soak window of synthetic checks (Gatus probes + trace-level health), the same image auto-promotes to uat. Promotion uat → prod requires an explicit human approval recorded in the deployment system - the auditable change-control gate.

The image promoted to prod is byte-identical to the image that passed uat. Environment differences are configuration (OpenBao-issued secrets, GrowthBook environment, ConfigMaps), never separate binaries.

Feature flags (GrowthBook for runtime gating; decouples deploy from release)

Deploy and release are separate operations. Code ships dark behind a GrowthBook flag with a safe default, then flips on - per environment, per cohort, or globally - once observability confirms the new path is healthy. A risky change can be merged, deployed, and verified in prod with traffic still on the old path before any user sees the new behaviour.

• Flags are read at the call site with a default= matching the historically-correct production value, so a GrowthBook outage degrades to pre-flag behaviour.
• Full reversal is one UI click with ≤30 s propagation.

Any PR that changes user-visible behaviour must either be flag-gated or carry a written justification for not being.

Schema migration safety (forward + backward compatible; lock budgets; multi-stage for breaking)

Every migration must be forward and backward compatible with the running version so that old and new code coexist during a rolling deploy. Breaking changes are decomposed into a sequence of compatible steps.

• Online by default: nullable ADD COLUMN, index creation CONCURRENTLY, additive enum values. Run inline with deploy.
• Lock budget: a migration taking an ACCESS EXCLUSIVE lock for longer than the per-service budget is rejected and reshaped.
• Multi-stage for breaking changes: a column rename or non-nullable add ships as expand → backfill → dual-write → flip reads → contract, each as its own deploy.
• Reversibility: every migration ships with a tested down-path or a documented forward-fix. We do not run migrations whose only recovery story is "restore from backup".

Rollback strategy (forward-fix as default; rollback only on SEV-1; per-service runbook)

Forward-fix is the default response: ship the fix through the normal pipeline and let the flag or the next promotion carry it. This is feasible because deploys are cheap, CI is fast, and most risky behaviour is behind a flag.

• Flag flip - first lever for any feature-gated change. No deploy, ≤30 s.
• Forward-fix deploy - second lever for code-path bugs that are not flag-gated.
• Image rollback - reserved for SEV-1 (broad outage, data-integrity risk). Each service has a per-service runbook covering the rollback command, the migration-compatibility window, and on-call escalation.
• Database rollback is never automatic. Because migrations are forward+backward compatible, the previous image serves against the new schema. Schema reversal, when needed, is a separately-reviewed migration.

Release cadence (continuous deploy; release notes auto-generated)

We do not batch releases. Every merge to main is a candidate release; a typical service ships several times a day. Release notes are auto-generated from PR titles and the squash-commit log, grouped by service, and published on each prod deploy.

What we explicitly do NOT do

• Long-lived branches. No develop, no release/*, no months-old feature branches.
• Manual hotfixes that skip CI. No break-glass path bypasses lint, tests, or the security scan. A genuine emergency uses the same pipeline with a fast-tracked review.
• Deploys that require manual database surgery. A migration that cannot run unattended is reshaped into compatible stages before it merges.
• Environment-specific binaries. The image in prod is the image that passed uat. Differences are configuration, not code.
• Auto-promotion from uat to prod. The uat → prod gate is always a human decision, recorded for audit.