Evaluation metrics¶

triagesim.utils scores a finished run against its ground truth. Every function consumes only environment output — the artifact's trace, belief and ground_truth — so metrics can be recomputed offline from a saved JSON file, with no model calls and no agent internals.

The functions degrade gracefully: when something needed is missing they return None rather than raising, so a batch job does not die on one malformed run.

`compute_all_metrics`¶

The orchestrator. It is keyword-only:

from triagesim.utils import compute_all_metrics

metrics = compute_all_metrics(
    trace=artifact["trace"],
    belief=artifact["belief"],
    ground_truth=artifact["ground_truth"],
    expert_red_flags={"syncope", "hypoxia", "tachycardia"},  # optional
)

Parameter	Type	Required
`trace`	`list[dict]`	Yes
`belief`	`dict`	Yes
`ground_truth`	`dict`	Yes
`expert_red_flags`	`set[str] \\| None`	No

It returns a four-key nested dict:

{
    "triage": {...},           # triage_decision_metrics + first_correct_turn
    "belief_coverage": {...},  # belief_coverage_metrics
    "red_flags": {...},        # red_flag_metrics
    "explanation": {...},      # explanation_support_metrics
}

Note that time_to_first_correct_triage is not a separate top-level key — its value is merged into metrics["triage"]["first_correct_turn"].

`triage_decision_metrics`¶

triage_decision_metrics(trace, ground_truth) -> dict

Compares the final predicted triage level against ground_truth["acuity"].

Key	Type	Meaning
`ground_truth`	`int \\| None`	The reference acuity.
`final_prediction`	`int \\| None`	Last triage level in the trace.
`correct`	`bool \\| None`	Exact match.
`absolute_error`	`int \\| None`	`abs(pred - gt)` — how many levels off.
`over_triage`	`bool \\| None`	Treated as more urgent than truth.
`under_triage`	`bool \\| None`	Treated as less urgent than truth.

The sign convention¶

Both ESI and ATS run 1 (most urgent) to 5 (least urgent), so the arithmetic inverts the intuition:

error = pred - gt
over_triage  = error < 0   # predicted a LOWER number = MORE urgent
under_triage = error > 0   # predicted a HIGHER number = LESS urgent

A prediction of 1 against a true acuity of 3 gives error = -2, which is over-triage: the nurse escalated a patient who did not need it. The reverse — predicting 4 for a true acuity of 2 — is under-triage, the clinically dangerous direction.

Under-triage is the failure mode that matters

Over- and under-triage are not symmetric harms. Over-triage wastes resources; under-triage delays care for someone who is deteriorating. Report them separately rather than collapsing both into absolute_error.

If either acuity is absent from the ground truth or no triage level appears in the trace, every field except the first two is None.

How the final prediction is found¶

The trace is scanned in reverse. The first entry whose action.type is "end" supplies its triage level; otherwise the most recent action carrying a triage key wins. Since utterance and end actions record a triage level but check_vital and log_red_flag actions do not, this resolves to the last spoken or final decision.

`time_to_first_correct_triage`¶

time_to_first_correct_triage(trace, ground_truth) -> int | None

Returns the turn index at which the nurse first stated the correct level, or None if it never did — or if acuity is missing.

This measures efficiency separately from accuracy. Two runs can both finish correct while one took twelve turns and the other took three.

First, not final

The nurse may state the right level early, revise away from it, and land somewhere else. A run can therefore have first_correct_turn == 2 and correct == False. Read the two together.

`belief_coverage_metrics`¶

belief_coverage_metrics(belief) -> dict

Structural measures of how much was elicited. No ground truth is needed.

Key	Type	Meaning
`num_associated_symptoms`	`int`	Distinct associated symptoms in the belief state.
`num_red_flags_inferred`	`int`	Values under the `red_flags` belief slot.
`has_chief_complaint`	`bool`	Chief complaint was established.
`has_pain_location`	`bool`	Pain location was established.
`has_pain_severity`	`bool`	Pain severity was established.
`has_duration`	`bool`	Symptom duration was established.
`vitals_known`	`list[str]`	Which vitals were released.
`num_vitals_known`	`int`	How many, of the five available.

Useful for asking whether a correct decision was actually earned: a nurse that guesses the right acuity with num_vitals_known == 0 and no pain severity got lucky.

`red_flag_metrics`¶

red_flag_metrics(belief, expert_red_flags=None) -> dict

Scores the flags the nurse explicitly logged, read from belief["red_flags_logged"].

Without expert annotations you get three descriptive keys:

Key	Type
`num_red_flags_logged`	`int`
`logged_any`	`bool`
`logged_flags`	`list[str]`, sorted

Passing expert_red_flags adds set-comparison keys:

Key	Type	Meaning
`expert_red_flags`	`list[str]`	The reference set, sorted.
`true_positives`	`list[str]`	Logged and expected.
`false_positives`	`list[str]`	Logged but not expected.
`false_negatives`	`list[str]`	Expected but missed.
`precision`	`float \\| None`	`None` when nothing was logged.
`recall`	`float \\| None`	`None` when the expert set is empty.

Matching is exact string equality

"tachycardia" and "Tachycardia" are different flags to this function, as are "hypoxia" and "low oxygen saturation". The environment normalises case and whitespace when accepting flags, but it does not map synonyms. Normalise your expert set to lowercase, and expect free-text paraphrases to depress precision and recall in ways that reflect vocabulary rather than clinical judgement.

F1 is not computed. Derive it if you need it:

p, r = metrics["red_flags"]["precision"], metrics["red_flags"]["recall"]
f1 = 2 * p * r / (p + r) if p and r else None

`explanation_support_metrics`¶

explanation_support_metrics(trace, belief) -> dict

Key	Type	Meaning
`has_explanation`	`bool`	An explanation was found on the final nurse action.
`explanation_length`	`int`	Word count, or `0`.

This is a placeholder

The source marks this function as hooks-only, with faithfulness checks — citation overlap, evidence hallucination — listed as future work. It reads the explanation from the final nurse action's action dict, where explanations are not stored, so in practice has_explanation will typically be False even when the trace contains rich reasoning.

To analyse explanations today, read them from the trace entries directly:

explanations = [
    step["explanation"] for step in artifact["trace"]
    if step.get("explanation")
]

Aggregating across runs¶

Single runs are noisy — the models sample, so the same configuration produces different dialogues. Aggregate before drawing conclusions:

import json
import statistics
from pathlib import Path

from triagesim.utils import compute_all_metrics

EXPERT_FLAGS = {"syncope", "tachycardia", "hypoxia"}

rows = []
for path in sorted(Path("runs").glob("*.json")):
    artifact = json.loads(path.read_text())
    rows.append(
        compute_all_metrics(
            trace=artifact["trace"],
            belief=artifact["belief"],
            ground_truth=artifact["ground_truth"],
            expert_red_flags=EXPERT_FLAGS,
        )
    )

scored = [r for r in rows if r["triage"]["correct"] is not None]
errors = [r["triage"]["absolute_error"] for r in scored]

print(f"runs scored:   {len(scored)}/{len(rows)}")
print(f"accuracy:      {sum(r['triage']['correct'] for r in scored) / len(scored):.2%}")
print(f"under-triage:  {sum(r['triage']['under_triage'] for r in scored) / len(scored):.2%}")
print(f"over-triage:   {sum(r['triage']['over_triage'] for r in scored) / len(scored):.2%}")
print(f"mean abs err:  {statistics.mean(errors):.2f}")

turns = [r["triage"]["first_correct_turn"] for r in scored
         if r["triage"]["first_correct_turn"] is not None]
if turns:
    print(f"median turns to first correct: {statistics.median(turns)}")

Filtering on correct is not None matters: unscorable runs would otherwise be counted as failures and quietly bias the accuracy downward.

← PreviousArtifacts

Next →Audio

Evaluation metrics¶

compute_all_metrics¶

triage_decision_metrics¶

The sign convention¶

How the final prediction is found¶

time_to_first_correct_triage¶

belief_coverage_metrics¶

red_flag_metrics¶

explanation_support_metrics¶