triagesim.utils

Evaluation metrics computed from a run artifact.

from triagesim.utils import compute_all_metrics

Every function consumes only environment output, so metrics can be recomputed offline from a saved artifact. See Evaluation metrics for the meaning of each returned key and for the over/under-triage sign convention.

Orchestrator

compute_all_metrics 

compute_all_metrics(
    *,
    trace: List[Dict[str, Any]],
    belief: Dict[str, Any],
    ground_truth: Dict[str, Any],
    expert_red_flags: Optional[Set[str]] = None,
) -> Dict[str, Any]

Compute all metrics for a single simulation run.

Triage decision

triage_decision_metrics 

triage_decision_metrics(
    trace: List[Dict[str, Any]],
    ground_truth: Dict[str, Any],
) -> Dict[str, Any]

Evaluate final triage decision.

Returns:

TypeDescription
Dict[str, Any]
  • correctness
Dict[str, Any]
  • absolute error
Dict[str, Any]
  • over/under-triage flags

time_to_first_correct_triage 

time_to_first_correct_triage(
    trace: List[Dict[str, Any]],
    ground_truth: Dict[str, Any],
) -> Optional[int]

Turn index when correct triage was first stated.

Belief coverage

belief_coverage_metrics 

belief_coverage_metrics(
    belief: Dict[str, Any],
) -> Dict[str, Any]

Structural metrics of what information was elicited.

Red flags

red_flag_metrics 

red_flag_metrics(
    belief: Dict[str, Any],
    expert_red_flags: Optional[Set[str]] = None,
) -> Dict[str, Any]

Metrics for explicitly logged red flags.

Works even without expert annotations.

Explanations

explanation_support_metrics 

explanation_support_metrics(
    trace: List[Dict[str, Any]], belief: Dict[str, Any]
) -> Dict[str, Any]

Placeholder for explanation faithfulness metrics.

Future ideas: - Citation overlap - Evidence hallucination checks

Text control

enforce_response_budget 

enforce_response_budget(
    text: str, response_length: str
) -> str

Trims patient utterance ONLY if it clearly violates the persona response_length budget.

← PreviousPersonas
Next →Audio