Testing

Olly has five test layers. The first three are Go and run in CI; the triage LLM-eval and the Playwright browser suites run manually.

Layer	Where	Runner
Go unit & integration	services/<svc>/internal/**/*_test.go	go test ./...
Go HTTP E2E	tests/e2e/	go test ./<svc>/...
Go atomicity / outbox	tests/e2e/atomicity/	go test -run TestAtomicity_<Svc>
Triage LLM-eval E2E	services/triage/tests/e2e/	uv run pytest tests/e2e
Playwright browser E2E	each app repo's e2e/	npx playwright test

All Go commands set GOWORK=off and call /usr/local/go/bin/go directly (matching CI).

Go unit & integration

Each service keeps its tests beside the code under services/<svc>/internal/**/*_test.go (handler, service, repository, kafka packages). The densest suites are billing (23 files), enrollment (17), notifications (16), claims (13), policy-admin (13).

cd /root/olly/services/<svc>
GOWORK=off /usr/local/go/bin/go test ./... -v -timeout 5m -count=1

CI runs these per service in .github/workflows/unit.yml.

Go HTTP E2E

tests/e2e/ (module github.com/olly/e2e) hits a live service stack over HTTP. There is a package per service (claims, eligibility, enrollment, provider, notifications, triage, care, broker_api, group_scheme, member_portal, boundary) plus a journey suite. Shared auth/setup is in tests/e2e/setup/ and tests/e2e/helpers/. Base URLs come from env (CLAIMS_BASE_URL=http://localhost:4001, TRIAGE_URL=http://localhost:4008, Keycloak http://localhost:8093).

cd /root/olly/tests/e2e
GOWORK=off /usr/local/go/bin/go test ./claims/... -v -timeout 90s -count=1

CI runs these in e2e.yml, gated by a journey job (TestFullMemberJourney / TestAuthFlow / TestAllServicesHealthy). One job (claims lifecycle scaffold) is deliberately allowed to fail; the rule-driven adjudication engine is not built yet.

Go atomicity / outbox

A separate module tests/e2e/atomicity/ (own go.mod) runs the outbox failure-mode matrix (E1-E10) against a live stack, one job per service adapter (claims, billing, care, consent, enrollment).

cd /root/olly/tests/e2e/atomicity
GOWORK=off OLLY_KAFKA_BROKERS=localhost:29092 \
  /usr/local/go/bin/go test -v -run 'TestAtomicity_Claims' -timeout 5m -count=1 ./...

CI: atomicity.yml, with an Atomicity Gate that PRs should require.

Triage LLM-eval (the AI harness)

This is the headline AI test layer, at services/triage/tests/e2e/. Each scenario is a self-contained JSON in scenarios/ (334 files: v1/v2/v3/v5). test_scenarios.py auto-discovers them and dispatches by JSON version to the right runner. Every scenario is graded at four layers at once:

1. Content: what the assistant said (contains_any / excludes / regex / disposition_band).
2. Provenance: which tools the agent called and in what order, read from the Langfuse trace.
3. Persistence: what landed in triage.sessions / turns / judgements.
4. Cross-cutting: latency budget, an LLM tone_judge, and a Langfuse cost cap (advisory by default).

cd /root/olly/services/triage
TRIAGE_DSN="postgres://olly:olly@10.0.1.2:5432/triage" TRIAGE_URL="http://127.0.0.1:4008" \
  uv run pytest tests/e2e -v          # add -k RF-001 for one scenario

For stable verdicts despite LLM stochasticity, the wave runners take a median of N runs:

uv run python tests/e2e/scripts/run_v2_wave.py 'MVP-BP-*.v2.json' --parallel 3 --runs 3

Scenarios trace back to Olly_Triage_Test_Strategy_v1.0.docx.md (the RF/RU/MSK/PF/MH/LC/SB/EX test-ID series, P0-P2). Two skills automate authoring: triage-test-writer (ID → grounded JSON) and triage-test-validator (a five-layer check). The eight fixture users are gender/age personas (alice_5 … hugh_70); the NHS pathway DB has a per-(sex, age) variant tree, so the fixture user selects which tree the agent can open. Keys live in services/triage/fixture_api_keys.env.

Triage is a Python service, so its non-eval unit tests are pytest files directly under services/triage/tests/ (uv run pytest). The triage job in unit.yml runs go test and is a no-op for the Python suite.

Playwright (browser E2E)

The frontend suites live in their app repos, not in tests/playwright/:

• Employer app: /root/olly-employer-app/e2e/ (dashboard, quote-flow); npx playwright test (auto-starts the dev server). Vitest is the unit runner.
• Member app: /root/olly-lovableapp-test/e2e/ (9 numbered specs: login, home, provider-booking, claims, plan-utilisation, notifications, triage-journey, plus auth.setup.ts).
• web-admin: /root/olly/apps/web-admin/e2e/ (test:e2e), with a Keycloak auth.setup.ts that persists storageState.
• Ontology: /root/olly/tests/playwright/ontology/ (smoke, screenshots, concept-pages) against ontology.dev.hiolly.com.

Test users

The universal Keycloak password is Olly2026 (set via make keycloak-set-passwords), realm olly. Go E2E and atomicity use member@test.olly / provider@test.olly / admin@test.olly; the member app logs in as member@test.olly; the employer test user is hr@olldemo.co.uk (org_locator SCH-2026-000001).

CI

GitHub Actions on self-hosted runners labelled [self-hosted, olly] (olly-ci, dev-2, uat). Three workflows: unit.yml, e2e.yml, atomicity.yml. There is no CI job for the triage LLM-eval or the Playwright suites; those are run locally.