POC #4 - FINDINGS
Verdict: GO at target scale. The composed "open claim detail" path (RLS read → OPA Φ → vault lookup → Transit unwrap → audit write) meets the +50ms p99 budget at the ~300 req/s anchor. The first thing to saturate under overload is OPA-over-HTTP, not RLS, the vault, or crypto - fixable by sidecar/cache/embed.
Setup: real composed path against the running stacks - rls-poc postgres (via pgbouncer, the proven 1-statement RLS function), composed-poc OPA (field policy), transit-poc OpenBao (batch unwrap of 10 wrapped DEKs), synchronous audit insert. Go harness, pgx.
1. End-to-end latency (ms)
| stage | c=16 p99 | c=64 p99 | c=200 p99 |
|---|---|---|---|
A RLS read (claim_detail) | 3.81 | 4.78 | 12.96 |
B OPA Φ field decision | 3.41 | 17.43 | 61.34 |
| C vault token lookup (×10) | 2.14 | 5.12 | 14.43 |
| D Transit batch unwrap (×10) | 3.37 | 9.97 | 24.94 |
| E audit write (sync) | 3.26 | 6.39 | 15.91 |
| TOTAL | 9.62 | 29.66 | 87.62 |
| within +50ms p99? | ✅ | ✅ | ✗ |
Throughput plateaus ~4.4-4.9k req/s (≈16× our ~300 req/s peak); c=200 is well past the anchor and only there does the budget break.
2. Where the time goes
- At target load (c=16): every stage <1.4ms p50; the audit write is the largest single stage (1.35ms), crypto/unwrap is ~0.74ms. Composition is cheap.
- Under overload (c=200): OPA dominates - p50 17.8 / p99 61ms - while Postgres stages (RLS read, vault lookup, audit) stay single-digit-to-teens. A single shared OPA over HTTP is the bottleneck, not the data or crypto layers.
3. Directives
- Composition is fine at scale anchor - keep the sequence; no layer needs cutting at ~300 req/s.
- Don't share one OPA over HTTP. It saturates first. Run OPA as a sidecar (loopback, per service), enable decision caching, or embed the policy (OPA Go lib / WASM) to drop the HTTP hop. Matches the access doc's "sidecar per service".
- Audit write is the largest cheap stage and grows under load. Keep it synchronous on the detail path (low volume by I4); if detail-view QPS climbs, batching/async-with-durability is the next optimization - at the cost of T3 attribution latency.
- RLS read / vault lookup / Transit unwrap are each cheap; batch the unwrap (I4, done here).
4. Scorecard
| Threshold | Result |
|---|---|
| end-to-end added p99 < +50ms on "open claim detail" @ target | ✅ 9.6ms (c=16), 29.7ms (c=64) |
| identify the binding constraint | ✅ OPA-over-HTTP at overload (61ms p99 @ c=200) |
| each-layer-passes-but-sum-fails caught? | ✅ sum is fine at anchor; only fails at 16× peak, attributable to shared OPA |
5. Caveats
- Synthetic: all 10 unwraps use the same sample ciphertext (latency-identical); single table;
employer_adminrole only. Models cost, not correctness ofΦ. - One shared OPA instance - the realistic sidecar topology would lower B materially; the c=200 number is therefore pessimistic for OPA.
- Audit insert competes for the same pgbouncer pool as the RLS read; a dedicated vault DB (per the design) would decouple them.
Reproduce / source files
cd /root/composed-poc && docker compose up -d # then run the harness (see README)Source (raw):
