Triage File Intake (F6)

Design / contract — backend landed, not yet on main

This documents the contract for Feature F6 (#1043). The backend is up for merge in OllyInsurance/olly PR #1204 (branch feat/triage-file-intake-main) and is deployed live + reachable now at https://triage.dev.hiolly.com/triage-api (the frontend's TRIAGE_BASE) for integration — but it is not merged to main yet (currently a bind-mount overlay on the running dev service; merge #1204 to make it durable). Frontend wiring is OllyInsurance/app#60; integration is tracked in app#87. When F6 ships, fold the as-built surface into /services/triage, add the paths to the static Scalar spec api/openapi/triage.yaml, and enrich the FastAPI route decorators. (Scalar serves the static spec, not the live /openapi.json — both must be edited by hand.)

Model

Video is a layer on top of text. A clip / voice note is uploaded, one server-side multimodal call turns it into text (a transcript + two summaries), that text is mapped onto the existing v5 capture fields, and the narrower / routing stay text-only and unchanged. The cards[] array is server-driven UI the client renders. Video is the first kind; the same path generalizes to voice / photo / document.

record ─▶ POST /v5/media ─▶ asset_id
                 └▶ POST /v5/media/{id}/analyze ─▶ {triple, extracted_fields}
                          └▶ write-through to the v5 capture ─▶ routing
                          └▶ GET/PATCH /v5/sessions/{id}/summary ─▶ editable card

Endpoints

Method · Path	Purpose
POST /v5/media?kind=video|voice|photo|document	Store an uploaded clip → asset_id
GET /v5/media/{asset_id}	Fetch stored bytes (playback)
POST /v5/media/{asset_id}/analyze	Clip → transcript + capture fields (text-projection)
GET /v5/sessions/{id}/summary	Field-segmented editable summary
PATCH /v5/sessions/{id}/summary	Persist one member field edit

{id} / session_id is the chat's session UUID (chat_id).

POST /v5/media

Raw binary body (no multipart). kind from ?kind= (video|voice|photo|document, wins) or inferred from Content-Type (video/*→video, audio/*→voice, image/*→photo, application/pdf →document); missing kind and unknown Content-Type ⇒ 422. Optional headers X-Duration-Ms, X-Filename. Max upload 64 MB at the /triage-api gateway (nginx client_max_body_size, proxy_read_timeout 300s for the long analyze); the service itself caps at 250 MB (TRIAGE_MEDIA_MAX_BYTES). Response { asset_id, kind, duration_ms, size, content_type }.

POST /v5/media/{asset_id}/analyze

Request { "session_id": "<uuid>" } → Response:

{ "asset_id": "…",
  "triple": { "frame_summary": "…", "video_summary": "…", "audio_transcript": "…" },
  "extracted_fields": { "presenting_complaint": "…", "anatomical_region": "spine", … } }

The multimodal call runs server-side with the service's own GEMINI_API_KEY (callers send no key of their own); clips over ~18 MB route through the Gemini Files API, smaller ones inline (the inline request cap is ~20 MB, so a real phone clip would otherwise fail-soft to an empty triple). Because the client sets Content-Type from blob.type — often application/octet-stream on a native file:// URI, which Gemini rejects → silent empty triple — the analyzer sniffs the real MIME from the container magic bytes + kind before the call (v5_media_analyze.py _resolve_mime/_sniff_mime). Side effects (fail-soft): extracted_fields are written through to the live routing capture (Valkey triage:v5:fields:<id>) and source-tagged video in the durable summary store.

GET / PATCH /v5/sessions/{id}/summary

GET → { session_id, fields:[ {key, label, value, source, locked, captured, block}, … ] } (durable edit > live LLM capture > "not mentioned"). PATCH body { "key": "...", "value": ... } (note key, not field_key) → { session_id, field:{…} }. A locked:true field (engine red-flag) returns 409 — members cannot silently downgrade it (D-39 #1137).

Capture schema

analyze and the summary read/write the 13-field v5 capture set (V5_COUNTED_CAPTURE_FIELDS, services/triage/src/triage/v5_narrow.py:200) + presenting_complaint, carried turn-to-turn via the x-triage-v5-capture-b64 header (v5_chat_completions.py:252). Fields: presenting_complaint, anatomical_region, onset, duration, trigger, symptom_severity, symptom_character, radiation, function_impact, sleep_impact, work_impact, prior_episodes, self_treatment, patient_context. Each carries provenance source ∈ member | llm | video | engine.

Client media types & coverage

The client sends the raw recorded bytes with Content-Type = the platform's blob.type (no per-kind override, no multipart, no filename in the body — hints ride as X-Duration-Ms / X-Filename headers). Only video and voice are wired today (photo / document are in the MediaKind union but stubbed — no capture UI, no expo-image-picker / expo-document-picker dependency). Web voice never uploads — it uses the Web Speech API and sends the transcript as chat text. Every cell below was verified end-to-end through the live /triage-api against the real Gemini key:

Client	Kind	Default MIME (codec)	Analyze
iOS	video	video/quicktime (HEVC/H.265, .mov)	✅
Android	video	video/mp4 (H.264 + AAC)	✅
Web	video	video/webm (VP8/VP9 + Opus; Safari → video/mp4)	✅
iOS / Android	voice	audio/mp4 · audio/x-m4a (AAC .m4a)	✅
any	any	application/octet-stream (unresolved blob.type)	✅ (sniffed)
Web	voice	audio/webm (Opus)	❌ Gemini rejects — but not uploaded (Web Speech → text)
iOS	photo	image/heic	⬜ unwired (Gemini accepts HEIC/HEIF when built)
all	document	application/pdf	⬜ unwired (only PDF + text/plain analyzable)

MIME handling. The analyzer trusts an explicit, concrete Content-Type; otherwise (missing / octet-stream) it recovers the real type from the container signature + kind (_resolve_mime / _sniff_mime). Any type Gemini still can't decode degrades fail-soft to an empty triple — the deterministic heuristic (presenting_complaint + anatomical_region) still runs on whatever text came back.

Gemini Developer API accepted inputs (what GEMINI_API_KEY targets — stricter than the Vertex / Firebase AI Logic surface, which additionally accepts audio/mp4, audio/webm, video/quicktime):

• image png, jpeg, webp, heic, heif · video mp4, mpeg, mov, avi, x-flv, mpg, webm, wmv, 3gpp
• audio wav, mp3, aiff, aac, ogg, flac — empirically audio/mp4 / .m4a also transcribes; audio/webm does not · document application/pdf, text/plain

Open items: web voice, if it ever uploads the blob, must send kind=video (video/webm is accepted) or transcode Opus→OGG; photo / document capture is unbuilt (when built, set a real Content-Type — iOS HEIC + PDF/text/plain only); and analyze returns 200 + an empty triple on an undecodable type with no error signal, so the FE can't yet distinguish "silent clip" from "unsupported media" (candidate follow-up: a contract flag).

cards[] response protocol (D-36 #1134)

Server-driven UI cards ride in the response body under triage_extras.cards — the same channel as the wearable streams_events chips, and a peer of the SSE stage_frame protocol (see services/triage/src/triage/STREAMS.md). message.content is dual-emitted as the OpenAI-compat fallback, so a client that ignores cards still renders.

{ "type": "options|record|summary|disposition", "id": "card-…", "version": 1,
  "payload": { … }, "actions": [ { "action_id": "…", "type": "postback|edit", "label": "…" } ] }

• options — payload.options[] = {id, index, text, pw_hint?}; one action per option {action_id:"opt-<i>", type:"postback", …}.
• record (A6.1) — payload.fields = capture blob; action edit-record.
• summary (A6.5) — payload.{title, body, fields?}; action edit-summary.
• disposition — payload.{title, is_terminal, care_type?, advice?}.

Postback: a tapped option posts back the id, not the text — {card_id, action_id} in the request body or X-Triage-Card-Id / X-Triage-Action-Id headers. The server resolves action_id (opt-<i>) → the chosen option index and runs the narrower on that id (closes the #157/#158 free-text-advance hazard). Per-type version lets a client card-registry degrade an unknown version to content.

Status & references

• Stories: A6.1 #1055 · A6.2 #1126 · A6.3 #1127 · A6.4 #1128 · A6.5 #1130 · A6.6 #1132.
• Deferred decisions gating go-live: D-39 (#1137, scope & trust) · D-40 (#1138, consent) · D-41 (#1139, retention & at-rest encryption).
• Backend: PR #1204 (branch feat/triage-file-intake-main), deployed + reachable at triage.dev.hiolly.com/triage-api; verified against a real 23 MB clip via the Files API. Frontend wired + typecheck-clean but with open runtime gaps — see app#60; integration tracked in app#87.