Skip to content

Evaluation

Dr. Tro's approval of the active registry and current mapping review decisions is recorded in evaluation/clinical_review/approval_manifest.json. That completes clinical review for the approved registry. A reproducible evaluation report is still required before claiming that a particular deployment build meets the numeric release gate; see Clinical validation.

MIMIC-IV real-world laboratory corpus

MIMIC-IV v3.1 can provide realistic local laboratory labels, units, values, and reference ranges through labevents.csv joined to d_labitems.csv. It is first used as an unlabeled retrieval/context corpus. Because the v3.1 labevents LOINC field is not a reliable gold source, the project requires expert item-to-LOINC review before any row enters the gold set. See the MIMIC workflow.

MIMIC rows must be split by local itemid and label family. Randomly splitting individual observations would leak repeated labels and inflate accuracy. MIMIC should complement, not replace, a fax-derived holdout because it does not represent OCR corruption or report layout noise.

Report separately:

  • Auto-accepted precision.
  • Top-1 candidate accuracy, measured before the confidence gate.
  • Accepted exact accuracy, measured only among emitted mappings.
  • Top-3 recall.
  • Coverage.
  • Abstention rate.
  • Dangerous false-positive rate.
  • Confidence calibration.

The production target is at least 95% precision among auto-accepted mappings and at least 85% coverage on a frozen, clinically reviewed, multi-laboratory holdout. Coverage must be reported alongside precision so the system cannot appear accurate by abstaining on nearly everything. These are release gates, not claims that may be inferred from an engineering stress CSV.

The evaluator must not use the emitted loinc_code for candidate top-1 accuracy because abstained results intentionally have no emitted code. Each mapping result also records stage counts in provenance so retrieval loss, safety rejection, ranking loss, and confidence-gate abstention can be measured separately.

Multi-Laboratory Holdout

The holdout must preserve laboratory source and split by source laboratory as well as alias family. At least one evaluation split must contain laboratories that were never used to build registry evidence. For clinically ordinary rows, compare results using the original source and an unseen-source label; a source-specific record may boost its own laboratory but must not be the only route to the correct code.

Report these slices separately:

  • known-source rows with source evidence available.
  • known-source rows with only universal retrieval available.
  • unseen-source laboratory holdout.
  • synonymy, OCR noise, specimen, time, method, property/unit, and panel/class mismatch categories.

The source-neutral software fixture covers Creatinine, Urine, FSH, CRP, Albumin CSF, and 24-hour urine patterns with an original and unseen source name. It is a regression check, not an independent multi-laboratory holdout. Clinical review of the active registry is complete; the remaining release evidence is the saved metric report for the frozen holdout.

v2 Engineering Set Interpretation

The v2 engineering file is useful for stress testing retrieval and OCR, but it does not contain source-laboratory holdout fields. Its repeatable label audit flags 126 of 425 rows: 89 contain Calculated while the expected term is non-calculated, and 37 specify serum/plasma while the expected term uses whole blood. These flags identify explicit-axis cases that should remain visible in regression review; they must not be silently treated as model errors or used to tune the release threshold.

Historical v2 figures are engineering diagnostics only. The file contains contradictory labels and must not be used as a clinical claim or as a threshold tuning target. The current release gate is a frozen, expert-reviewed holdout; the v2 set remains useful for regression and latency diagnostics.

Write per-row diagnostics with:

python -m loinc_mapper evaluate `
  --input evaluation/stress_test/stress_test_output.csv `
  --umls-path assets/umls/2026AA `
  --scispacy-model en_core_sci_md `
  --vector-index assets/loinc/2.82/terms.faiss `
  --vector-metadata assets/loinc/2.82/terms.json `
  --details evaluation/stress_test/production_detailed_report.json

Always inspect the saved provenance. Production FAISS evidence should include semantic_index.backend=faiss, a nonzero semantic_retrieval_candidate_count, and the pinned LOINC/model metadata.

The current stress CSV is a catalog-verified engineering dataset, not an independent multi-laboratory holdout. Its negative LDL Direct row is an adversarial assertion that must not be evaluated as an ordinary expected-code row: other rows with the same explicit method expect direct-assay code 18262-6.

Profile Matrix and Performance Gate

The same frozen rows should be evaluated through the production path and every controlled ablation. This answers questions such as "did UMLS help?" or "did FAISS improve coverage?" with metrics rather than intuition:

python -m loinc_mapper evaluate `
  --input evaluation/gold/holdout.csv `
  --details evaluation/gold/profile_matrix.json `
  --profile-matrix `
  --umls-path assets/umls/2026AA

The report contains:

  • full production pipeline, with the registry fast path disabled;
  • full production pipeline with the safe fast path enabled;
  • no UMLS;
  • no FAISS;
  • no UMLS and no FAISS; and
  • deterministic-only diagnostic mode.

For each profile it records accepted precision, top-1/top-3, coverage, abstention, dangerous false positives, per-stage mean/p50/p95 latency, and fast-path equivalence. The evaluation profile cannot produce an Elation payload. A change is not promotable if it loses a known-correct regression result, reduces measured accepted precision or unseen-source coverage, or adds a dangerous false positive.