Observability & Telemetry

Position

Olly treats observability as a product surface, not a debugging afterthought: every Go service and the triage agent emit OpenTelemetry over OTLP, every signal lands in a single backend per type, and every entry on the status board is a Gatus probe with a named owner. The boundary - we instrument to answer "what is the platform doing right now?", never so deeply that PII, secrets, or raw clinical prompts leave the trust boundary inside a span.

OpenTelemetry as the standard (every Go service + the triage agent emit OTLP)

OTel is the only telemetry contract we accept. Go services use the official Go SDK; the triage PydanticAI agent uses the Python SDK with auto-instrumentation. Emitters ship OTLP to a single collector tier; nothing writes directly to a backend.

• Collector: otel-collector on dev-2 receives OTLP gRPC on 10.0.1.2:4317, applies resource attribution and tail-sampling, and fans out to Tempo / Loki / Prometheus. Grafana Alloy runs alongside as host-metrics + log scraper into the same pipeline.
• Required resource attributes: service.name, service.version, deployment.environment, git.sha, host.name. Spans missing these are dropped at the collector.
• Trace propagation: W3C traceparent is mandatory on every HTTP and Kafka edge. APISIX injects it at the gateway; the triage agent threads it through PydanticAI tool calls.

A quote bind fanning out across enrollment, billing, and provider shows up as one trace, not five.

Storage backends (Tempo for traces; Loki for logs; Prometheus + Grafana for metrics)

One backend per signal type, all queryable through a single Grafana on dev-2. Jaeger is a read-only fallback.

Signal	Backend	Retention	UI
Traces	Tempo	14 days hot	Grafana → Tempo
Logs	Loki	30 days hot	Grafana → Loki
Metrics	Prometheus	30 days hot, 15 mo downsampled	Grafana
Synthetic checks	Gatus	90 days rolling	status.dev.hiolly.com

Logs are structured JSON; every line carries trace_id, so jumping from a Grafana panel to the originating trace is a click. RED is mandatory per HTTP handler; USE per datastore client.

LLM tracing (Langfuse captures every triage tool call; queried by sessionId)

A single triage turn fans into LLM calls, tool calls, pathway lookups, and retries that are not legible in a normal trace viewer. These flow into Langfuse at langfuse.dev.hiolly.com alongside OTel.

• Every session has a stable sessionId; every model call, tool invocation, and lever read is a Langfuse observation under it.
• Inputs/outputs are structured payloads, not raw transcripts (see boundary section).
• triage_session_debug(sessionId) on the MCP returns a merged session + turns + Langfuse view; engineers debug from that, not docker logs.

Langfuse does not replace Tempo - a triage request still produces an OTel trace covering the FastAPI handler and outbound HTTP.

Synthetic monitoring (Gatus per-endpoint health checks; on the status dashboard)

Real-user telemetry tells us what users hit; Gatus tells us what is reachable. Every endpoint we own - every *.dev.hiolly.com domain, every Go service /health, APISIX, Keycloak, OpenBao, Langfuse, Novu, OpenMetadata - has a Gatus probe with an explicit success contract (status, body match, TLS validity, latency budget). Probes render on the public status board so EY and engineers see the same reality.

Synthetics catch what RUM cannot

A page can return 200 while a dependency is silently degraded. Gatus probes the dependency directly.

Sampling strategy (100% in dev; head-based + tail-based in prod)

Sampling is environment-aware and lives in the collector, never in service code.

• Dev: 100% trace sampling. We trade storage for the ability to walk any failed request.
• UAT and prod (forward): head-based at the SDK (10% baseline) plus tail-based in the collector, which always keeps (a) any trace with an error span, (b) any trace exceeding a per-route latency SLO, (c) any trace flagged by an OTel baggage key (canaries, support escalations).
• Triage traces stay at 100% in every environment until v5 stabilises.

Alert philosophy (SLO-burn alerts over threshold alerts; every alert links to a runbook)

We do not page on raw CPU, memory, or error counts. We page on error budget burn against published SLOs.

• Each user-facing capability (quote bind, claim submit, triage turn, login) has an availability and latency SLO in a slo.yaml next to the service.
• Prometheus computes burn-rate alerts at two windows (1-hour fast, 6-hour slow) using the SRE multi-window multi-burn-rate recipe.
• Every alert carries three required labels: severity (SEV-1..3), runbook (docs URL), owner. Missing labels fail the CI linter on the rules repo.
• Threshold alerts (disk full, cert expiring, broker lag) survive only for things not naturally SLO-shaped and still require the same labels.

On-call response (rotation; SEV severity rubric; escalation path)

A single rotation covers the whole platform - an on-call carrying the user journey end-to-end produces better triage than three specialists each seeing a fragment.

Severity	Definition	Initial response	Escalation
SEV-1	User-visible outage of a core flow	Ack ≤ 5 min; incident channel opened	Secondary at 15 min
SEV-2	Degraded capability with workaround, or non-core flow down	Ack ≤ 30 min	Secondary at 1 h
SEV-3	Background regression, no user impact	Next business day	None

Every SEV-1/2 closes with an incident note linked from the runbook; the runbook is updated with whatever the on-call wished they had known at 03:00.

What we explicitly do NOT observe (PII in spans; secret values; raw triage prompts - we redact or hash)

Some things never enter the telemetry plane, and the collector enforces this rather than relying on every service author to remember.

• PII in spans and logs: names, DOBs, addresses, phone, email, NHS numbers, and free-text symptom descriptions are stripped or hashed by the collector's attribute-processor before reaching Tempo or Loki. Member correlation uses party_locator, an internal non-reversible handle.
• Secret values: tokens, API keys, Bao leases, signed URLs, Stripe secrets, Keycloak passwords match a deny-list of keys plus a regex pass over values, and become the literal string [REDACTED].
• Raw triage prompts and user utterances: Langfuse stores the structured input the agent acted on (pathway, facts, lever values, model, token counts) - never the raw message verbatim. A separate access-controlled clinical-evidence store handles full transcripts for safety review.
• Third-party PHI: NHS pathway content, EDI 270/271 payloads, and partner API responses are treated as PHI at the boundary and redacted on the same rules.

Negative space, on purpose

If a value is sensitive, it should be impossible to find in Grafana - not "unlikely". Collector-side redaction is the enforcement; service-side discipline is the defence-in-depth.