Skip to content

Pipeline Walkthrough

This tutorial explains one observation from the first character we receive to the LOINC code we either emit or reject. It is written for learning, not just for operating the CLI.

The production flow is:

raw OCR row
  -> LabObservation
  -> text and unit normalization
  -> AxisFactExtractor and UniversalSemanticIndex
  -> reviewed registry lookup
  -> deterministic LOINC/Part retrieval
  -> ScispaCy mention and abbreviation processing
  -> local UMLS exact bridge retrieval (derived FTS/character indexes when provisioned)
  -> candidate union
  -> six-axis and UCUM safety validation
  -> SapBERT vector retrieval and reranking
  -> feature ranker, when a reviewed model is pinned
  -> confidence and margin gate
  -> MappingResult
  -> Elation payload, only when mapped

The notebook version of this lesson is notebooks/01_pipeline_walkthrough.ipynb in the repository root.

1. The Input Object

The mapper does not receive an unstructured Python dictionary internally. It receives a LabObservation:

LabObservation(
    raw_name="low density lipoprotein cholesterol",
    value="100",
    unit="mg/dL",
    specimen=None,
    specimen_code=None,
    collection_specimen=None,
    collection_specimen_code=None,
    method=None,
    time_context=None,
    scale=None,
    observation_domain="LAB_RESULT",
    loinc_class_hint="CHEM",
    panel_context="lipid panel",
)

The name and value usually come from OCR. Unit, specimen, method, and time context may be missing because fax reports are inconsistent. Missing context is not silently invented as fact. The mapper may infer a context tag from the name, but the result still records what happened.

For production integrations, use the canonical specimen enums when the main project can identify them and retain the OCR wording for auditability:

LabObservation(
    raw_name="Hemoglobin",
    value="12.0",
    unit="g/dL",
    specimen="blood",
    specimen_code="blood_unspecified",
    collection_specimen="Blood, Venous",
    collection_specimen_code="blood_venous",
)

specimen_code values are unknown, blood_unspecified, whole_blood, serum, plasma, serum_or_plasma, urine, cerebrospinal_fluid, saliva, stool, sputum, amniotic_fluid, synovial_fluid, pleural_fluid, peritoneal_fluid, body_fluid_unspecified, tissue, and other. collection_specimen_code values are unknown, blood_venous, blood_arterial, blood_capillary, urine_clean_catch, urine_catheter, urine_24_hour, and other. A contradictory enum/text pair raises an error. Bld is a LOINC System value, not a UI value.

2. Report Domain and Laboratory Class

'specimen' means the material actually analyzed. 'collection_specimen' is separate evidence such as 'Blood, Venous' from a report footer. A venous draw can produce whole blood, serum, or plasma, so do not copy collection text into the analytical specimen field unless the report explicitly identifies the tested material. If the report parser provides a collection-only value, pass it through unchanged:

LabObservation(
    raw_name="Glucose",
    value="97",
    unit="mg/dL",
    specimen=None,
    collection_specimen="Blood, Venous",
)

Every extracted row is positional. Do not merge rows by analyte name alone: two 'Calcium' rows can belong to different panels or specimen contexts. The main OCR adapter must preserve both rows, their values, units, page, and report identifiers before calling map_many().

Routing happens before broad candidate retrieval. The current package accepts laboratory results and limits unknown observations to LOINC CLASSTYPE=1. Pass a trusted class when the OCR/report adapter knows it:

LabObservation(
    raw_name="Hemoglobin",
    unit="g/dL",
    specimen="whole blood",
    observation_domain="LAB_RESULT",
    loinc_class_hint="HEM/BC",
    class_source="report_metadata",
    class_confidence=1.0,
)

An explicit trusted class is strong retrieval/ranking evidence, not a hard clinical filter. A panel such as CBC or CMP is softer context. LOINC CLASS is an arbitrary grouping, so a wrong OCR section must not reject a safe CRP, TSH, endocrine, or chemistry result. LAB_RESULT still limits mapping to laboratory CLASSTYPE=1 terms. If the domain is imaging, history, diagnosis, or another future domain, this laboratory mapper abstains rather than searching unrelated LOINC terms. See Category-aware routing.

3. Normalization

Normalization creates a matching form while preserving raw_name in the observation for auditability. It is not an ML model and it does not select a LOINC code.

For text, the current normalizer performs these operations in order:

  1. Unicode NFKC normalization.
  2. Case folding, so LDL.C and ldl.c behave alike.
  3. Replace & with and.
  4. Replace punctuation and separators with spaces.
  5. Collapse repeated whitespace.
  6. Apply reviewed OCR substitutions, such as ld1 -> ldl, g1ucose -> glucose, and tr1glycerides -> triglycerides.

Examples:

Input Normalized text
LDL.C ldl c
Low-densitylipoproteincholesterol low density lipoprotein cholesterol
TRlGLYCERIDES FASTING triglycerides fasting
Ca1culated LDL calculated ldl

Units are normalized separately. For example, mg / dL, mg per dl, and mg/dL become mg/dl. This makes unit comparison deterministic.

The unit parser is not a finite dictionary of medical units. Its core is a composable UCUM grammar: prefixes such as m, u, n, and k combine with base atoms and exponents to produce dimensions and conversion factors. The versioned config/unit_surface_rules.json contains only laboratory display conventions that are not safe to infer from grammar alone, such as K/uL meaning 10^3/uL and m/uL meaning 10^6/uL in common blood-count reports. Unknown atoms remain invalid and go to review instead of being silently accepted as a new unit.

The mapper also creates retrieval variants. For LDL Cholesterol (Calculated), variants include the original normalized form, a compact form, and a form with context markers such as calculated and serum removed. The context markers are removed only for retrieval; method and specimen are still inferred and checked later.

3A. Unit And Method Safety Cases

The same grammar handles common report surfaces such as 'gm/dL', 'mcg/dL', 'mcg/mg', 'mlU/mL', 'mclU/mL', and indexed eGFR units such as 'mL/min/1.73m*2'. These are surface and exponent repairs, not analyte-to-code maps. The parser records the canonical unit and physical dimension; the validator still compares that dimension with the candidate Property.

Year-specific methods remain evidence-sensitive. A report that says only 'CKD-EPI' cannot be treated as 'CKD-EPI 2021'; the validator rejects the year-specific '98979-8' candidate unless '2021' is present or a reviewed source-laboratory mapping supplies that discriminator. A non-year-specific candidate may remain available for review.

4. Generated Context Vocabulary

The name can carry useful context:

from loinc_mapper.context import ContextExtractor

ContextExtractor().extract("LDL Direct").method.value       # "Direct assay"
ContextExtractor().extract("Cholesterol, LDL, Measured").method.value  # "Direct assay"
ContextExtractor().extract("LDL meas.").method.value         # "Direct assay"
ContextExtractor().extract("LDL Calc").method.value          # "Calculated"
ContextExtractor().extract("LDL Plasma").specimen.value      # "plasma"
ContextExtractor().extract("Hgb Whole Blood").specimen.value # "whole blood"

The extractor is not built from a complete hand-written _METHODS or _SPECIMENS tuple anymore. At catalog-build time it generates axis vocabulary from active LOINC term fields and LOINC Parts. It also imports the versioned config/axis_context_aliases.json file for surface forms LOINC cannot know for every local laboratory. The production artifact stores this evidence in axis_values, axis_aliases, axis_fts, and axis_trigram.

The alias file is data, not hidden code. Each row records its source and weight; it can be reviewed, scoped to a laboratory, replaced at a release boundary, and audited. It provides context only. It never directly selects a LOINC code.

At runtime, a schema-6 SqliteLoincCatalog first checks the exact generated axis table, then uses axis_fts token retrieval and axis_trigram recovery for release values that are not in the small in-memory first-token bucket. The extractor still checks whether the returned alias actually appears in the OCR text, so FTS is retrieval evidence, not an automatic clinical assertion. A reviewed alias overlay is loaded after the release snapshot; this lets a reviewed interpretation such as measured -> Direct assay take precedence over an official Part surface form without rebuilding the full catalog.

Reports commonly use measured, meas., or quantified to contrast a direct assay with a calculated result. The signal becomes Direct assay context and must agree with the candidate LOINC method. If the report uses a laboratory- specific phrase whose meaning is uncertain, the correct result is review rather than guessing. More model compute cannot safely invent that missing meaning.

For an input, inspect all axis evidence instead of only the selected value:

from loinc_mapper.context import ContextExtractor

context = ContextExtractor(catalog).extract("Cholesterol, LDL, Measured")
print(context.values())
print(context.candidates["method"])
print(context.conflicts)

The important distinction is between context.method and context.candidates["method"]. The first is the selected best interpretation used for filtering. The second is the complete evidence list, including source, span, and confidence, so an engineer can see whether the result came from a reviewed surface alias, an active LOINC Part, a term field, or SQLite retrieval.

The selected method is only a filter. The final LOINC code is still chosen after candidate retrieval, unit validation, six-axis validation, SapBERT ranking, and the confidence gate.

Some report labels contain both a broad purpose and a specific method. For example, ANA Screen, IFA contains Screen and immunofluorescence (IFA). The generated vocabulary treats Screen as purpose context when a more specific analytical method is present, so it does not create a false method conflict. If Screen is the only method evidence, it remains available for screening terms. The exact approved ANA registry entry targets 42254-3, whose pinned-release method is IF, property is PrThr, class is SERO, and result eligibility is Both.

For Cholesterol, LDL, Measured, the earlier pipeline abstained because Measured was not extracted. UMLS and FAISS retrieved the direct-assay code, but calculated, methodless, and direct LDL siblings all survived safety with the same text/unit evidence. The confidence gate correctly found a zero margin. Once the generic method signal is extracted, calculated and methodless siblings are rejected before ranking, leaving the direct-assay candidate.

For this exact row, the trace should be read as follows:

Stage Evidence Safety meaning
Raw input Cholesterol, LDL, Measured, 133, mg/dL Preserve the original label and value.
Normalizer cholesterol ldl measured plus hypotheses Improve retrieval without selecting a code.
Context vocabulary Measured -> Direct assay Add a method discriminator from reviewed data.
SQLite/FTS LDL aliases and sibling codes Produce candidates, not a final answer.
UMLS/ScispaCy CUI/name evidence when available Expand meaning; never override LOINC.
UCUM validator mg/dL has mass/volume dimension Reject particle/substance candidates.
Six-axis validator Candidate method must be Direct assay Remove calculated and methodless siblings.
SapBERT/FAISS Rank the safe survivors Resolve remaining lexical similarity.
Confidence gate Check score and margin Emit only if confidence and separation pass.

If a row abstains, inspect the first stage that lost the intended evidence. A larger embedding model cannot repair a missing unit, an ambiguous specimen, or an absent method discriminator safely; those cases should become reviewed data or explicit report-context fields.

4A. Universal Facts Before Source-Specific Evidence

A laboratory can appear only once in a fax stream. The mapper therefore does not require a matching source_laboratory record for ordinary results. Before the registry, UMLS, or SapBERT has any say, AxisFactExtractor reads facts stated by the row and UniversalSemanticIndex retrieves compatible release terms.

from loinc_mapper.universal import AxisFactExtractor
from loinc_mapper.context import ContextExtractor

observation = LabObservation(
    raw_name="Creatinine, Urine",
    value="54.7",
    unit="mg/dL",
    specimen="Urine",
    source_laboratory="A laboratory we have never seen",
)
context = ContextExtractor(catalog).extract(observation.raw_name)
facts = AxisFactExtractor().extract(observation, context)
print(facts.to_dict())

The useful evidence is Component=Creatinine and System=Urine. That is not a manually maintained Creatinine, Urine -> code dictionary. The universal index uses the active local LOINC Components, Part links, consumer/related names, linguistic variants, and catalog aliases to retrieve urine siblings. The six-axis/UCUM gate rejects serum/plasma terms, 24-hour terms without a stated duration, and method-specific terms without a stated method.

Candidates are then grouped by an AxisSignature:

Component | Property | Time | System | Scale | Method

This stops a long list of near-identical aliases from creating a fake confidence tie. It does not collapse clinically different siblings. If two signatures are both safe and the report lacks the discriminator, the mapper abstains for review.

collection_specimen="Blood, Venous" remains collection provenance rather than an analytical System fact. It is compatible family evidence for whole blood or serum/plasma rows, but it must not turn a venous draw into a hard assertion that every result is serum, plasma, or whole blood. See Universal multi-lab mapping for the full design.

4. The Local LOINC Catalog

LoincCatalog loads active terms from the local LOINC release under data/Loinc_2.82/. Each term contains the six axes:

Axis Meaning Example
Component What is measured Cholesterol in LDL
Property What characteristic is measured Mass concentration (MCnc)
Time When or over what duration Point in time (Pt)
System Specimen or biological system Serum or plasma (Ser/Plas)
Scale Result scale Quantitative (Qn)
Method Analytical method when clinically meaningful Calculated

The release compiler builds SQLite FTS5 token retrieval and a character-trigram retrieval sidecar. Alias source and token document frequency are retained as evidence, so generic words such as level, serum, count, and chemistry are down-weighted by the release vocabulary instead of being deleted by a permanent hand-maintained list. The compiled schema uses an index on alias code so explanation and scoring do not scan the four-million-row alias table. The compiler also builds the generated axis vocabulary tables described above. It writes a temporary SQLite image and atomically replaces the release artifact only after completion. Older artifacts without axis tables use a catalog-term fallback and expose that fallback in provenance; they are not silently treated as equivalent to the generated schema. The catalog remains the authority for whether a code exists, is active, and has the expected axes.

5. Deterministic Registry and Candidate Generation

The registry is a reviewed, versioned mapping file at config/mapping_registry.json. It is intentionally human-editable, but it is not the only path to a result. The universal LOINC-derived path runs for every ordinary row, including an unseen laboratory.

Registry entries have two governed scopes:

  1. universal_template: an expert-confirmed, clinically invariant surface with explicit unit/axis constraints. It can assist every laboratory.
  2. source_evidence: a local test ID, vendor method, or display convention. It is used only for the exact source_laboratory and never suppresses universal retrieval.

For example:

{
  "alias": "LDLP",
  "target": "54434-6",
  "required_units": ["nmol/L"]
}

The registry does not mean that every matching alias is accepted. It creates a high-quality candidate. The candidate still passes unit and axis validation.

Clinician-approved entries may also carry a generated verified_aliases list. These are bounded normalized, compact, or reviewed OCR forms of the approved surface, not new semantic synonyms. An exact match to one of them can use the same replay-certified authority. Contained and arbitrary fuzzy matches remain normal candidates and still need the confidence and margin gate.

When one clinical name has more than one LOINC sibling, add one constrained entry per verified measurement rather than one unconstrained name map. For example, serum/plasma sex hormone binding globulin has an nmol/L molar concentration code (13967-5) and a ug/dL mass concentration code (2942-1). Free Androgen Index is represented by 24125-7 when the result is reported as %. These entries use the LOINC property and required unit as additional evidence, so a report cannot be routed to the wrong sibling merely because the displayed name is SHBG or Sex Hormone Binding Globulin. The aliases were taken from the pinned release's term/related-name evidence and must still be clinically reviewed before promotion.

The deterministic generator combines these sources:

  1. registry: reviewed alias with an exact or token-boundary-contained target.
  2. exact_alias: exact match to a catalog alias.
  3. token_retrieval: indexed catalog-token and IDF-weighted retrieval, with FTS5 used only as a bounded prefix fallback.
  4. character_retrieval: generated character-trigram/close-vocabulary retrieval for unseen OCR edits.

SapBERT sidecar retrieval is then added to the same candidate union. Candidates are deduplicated by code, but their registry, catalog, UMLS, and semantic evidence is retained.

The current fuzzy limit is 100. This is a retrieval limit, not a final answer limit. Registry and exact candidates are retained before safety validation. A contained match is token-boundary based: the reviewed alias VLDL matches VLDL Cholesterol Cal, but LDL does not match inside LDLP. The suffix Cal is generated reviewed context evidence for Calculated, so calculated and methodless siblings can be separated before ranking. Token candidates use information-content weighting so a specific token such as sodium matters more than a generic token such as level. Character retrieval runs once on the best compact lexical query and tests distributed 3-of-4 trigram intersections. A single OCR character error usually damages only one trigram, while the remaining intersections keep the database query bounded.

6. UMLS and ScispaCy

UMLS is a concept bridge, not the final authority for the code.

ScispaCy first processes the raw name, detects mentions, and expands abbreviations. The current adapter sends the raw name, detected entity text, short contiguous entity subphrases, and abbreviation long forms to the local UMLS SQLite index. Once build-umls-serving has completed, the preferred runtime backend reports serving_exact+fts5+trigram: it has exact normalized lookup, the active CUI-to-LOINC bridge, LOINC-linked FTS5 retrieval, and a character 3-gram recovery index. This is still candidate generation, not a final clinical decision. Local LOINC SQLite FTS5/trigram retrieval and the FAISS sidecar provide independent evidence that is retained alongside UMLS evidence.

The local index performs exact and normalized lookup and returns concepts containing:

CUI
preferred or matched name
linking confidence
source vocabulary
LOINC source-code evidence, when available

For example, several local names may converge on the same UMLS CUI. A CUI is not assumed to map one-to-one to a LOINC code. A broad concept such as glucose can be associated with many LOINC terms. The mapper therefore uses the UMLS concept name to retrieve local LOINC candidates, then applies units and axes before ranking.

The local LOINC release remains authoritative because:

  • UMLS concepts can be broader than a specific laboratory observation.
  • UMLS may contain many source vocabularies and many codes for one concept.
  • Elation requires a valid LOINC code, not just a CUI.

The production linker is configured with assets/umls/2026AA. The licensed SQLite files remain outside Git. Confirm the backend in every deployment smoke test and in MappingResult.provenance rather than assuming that a file with the expected name is complete.

7. Six-Axis and Unit Safety

Every candidate is checked before SapBERT is allowed to rank it.

The validator checks:

  • Unit against the candidate's example units and any registry-required units.
  • Property against the unit family. For example, a mass unit cannot support a molar property.
  • Specimen against the LOINC system.
  • Explicit method against the LOINC method.
  • Explicit duration against point-in-time terms.
  • Explicit scale against the candidate scale.

Some failures are hard failures. A hard failure removes the candidate from ranking. This is deliberate: a wrong unit such as mg/dL for LDL particle number is not merely a lower similarity score.

Example:

LDLP + nmol/L -> particle-number candidates remain
LDLP + mg/dL  -> particle-number candidates are rejected

An ambiguous observation may therefore have no accepted code even when text similarity is high. That is a safety feature, not a crash.

8. SapBERT Reranking

SapBERT is used after deterministic retrieval, UMLS expansion, and safety filtering. The prebuilt FAISS index searches the normalized vector sidecar at runtime, but the model does not re-encode all 100,000 LOINC terms for each row.

Before ranking, SapBERT also searches the required precomputed vector sidecar. For each validated candidate, the mapper builds stable bounded candidate text from:

long common name
short name
bounded aliases
component
property
time
system
scale
method

The model encodes the query and candidate texts into normalized vectors. Because vectors are normalized, the dot product is cosine similarity:

cosine(query, candidate) = query_vector dot candidate_vector

Current runtime parameters:

Parameter Current value Meaning
Model cambridgeltl/SapBERT-from-PubMedBERT-fulltext Biomedical embedding model
Batch size 32 Texts encoded per model batch
Max candidates 50 Candidates sent to transformer per row
Retrieval contribution 0.35 Deterministic evidence weight in the bootstrap score
Semantic contribution 0.65 SapBERT similarity weight
Registry bonus 0.15 Bonus after a registry candidate passes safety
Semantic hit limit 50 Sidecar hits retained per query variant
Semantic query limit 2 Raw/normalized hypotheses sent to FAISS per row

The untrained bootstrap score is approximately:

0.35 * retrieval_score
+ 0.65 * semantic_score
+ registry_bonus

The remaining retrieved tail is preserved for audit but receives score zero in the current bounded implementation. This keeps runtime predictable. The top 50 are selected by retrieval quality, not dictionary insertion order. When a candidate already has a FAISS similarity, the reranker reuses that precomputed similarity instead of encoding the same candidate text again. Candidates from deterministic or UMLS retrieval without sidecar evidence still receive direct SapBERT encoding. A pinned AdaRank-compatible or LambdaMART challenger consumes the same evidence as features; it is activated only after champion/challenger evaluation.

9. Confidence Gate

The top score alone is not enough. The policy checks:

confidence = top score
margin = top score - second score

Current defaults:

minimum_confidence = 0.82
minimum_margin = 0.08

The result is:

Status Meaning
mapped Candidate passed safety and confidence gates
abstain Evidence exists but confidence or separation is insufficient
invalid_candidate Candidates existed but all failed hard safety checks
no_candidate No candidate was generated

Only mapped results can produce test.loinc for Elation.

10. Reading a MappingResult

The result is designed for inspection:

result.status
result.loinc_code
result.confidence
result.margin
result.candidates
result.axis_evidence
result.unit_evidence
result.provenance
result.reasons

The candidate evidence tells us whether a candidate came from the registry, exact catalog alias, UMLS, token, character, or FAISS retrieval. SapBERT evidence contains the semantic similarity and the base retrieval score. Provenance records the LOINC release, registry version, linker/backend, ranker, model version, semantic-index manifest, query/hit counts, hard rejections, and stage counts. A production result must show semantic_index.backend=faiss and a nonzero semantic_retrieval_candidate_count; otherwise it was not measured with the full production retrieval path.

10A. Panel Context And Real OCR Failures

Panel context is a soft retrieval prior. A row from Comp. Metabolic Panel is more likely to be the routine serum/plasma chemistry observation than a urine or corrected-method sibling, so the generated panel rules add ranking evidence. They do not override an explicit specimen, unit, method, or LOINC axis conflict. The rules are data in config/panel_context_rules.json, not hidden code mappings.

For example, a CMP row is processed like this:

observation = LabObservation(
    raw_name="Calcium",
    value="9.2",
    unit="mg/dL",
    observation_domain="LAB_RESULT",
    loinc_class_hint="CHEM",
    class_confidence=1.0,
    panel_context="Comp. Metabolic Panel",
)
result = mapper.map(observation)

The mapper first rejects ratio candidates such as Calcium/Albumin because mg/dL is dimensional, rejects ionized/corrected candidates when those qualifiers are absent from the row, and then uses the panel prior to separate the standard total-calcium candidate. This is why the fix belongs in axis and unit safety, not in an LLM prompt.

Panel context does not turn a row into a panel. A row named eGFR cannot emit the separate Creatinine and eGFR predicted panel code merely because the report section is a CMP. Panel-class candidates are rejected unless the observed row name itself explicitly identifies a panel.

OCR units are repaired through the UCUM grammar when the repair is unambiguous: mgdL/ becomes mg/dL and gdL/ becomes g/dL. An unknown or dimensionally unsafe unit still causes review. The same principle applies to text: a close edit of a reviewed alias such as Albuminou can use the release-generated registry fuzzy lookup, while short risky aliases such as LDL are never fuzzy-expanded into LDLP.

Specimen wording has the same specificity rule. whole blood, serum, plasma, arterial blood, and venous blood are specific axis evidence. OCR text such as Type: Blood is broader: it permits blood-derived LOINC systems, including Bld and Ser/Plas, but does not permit urine or CSF systems. Treating generic Blood as equivalent to whole blood would wrongly reject routine serum/plasma laboratory results.

Blood, Venous needs the same care. It identifies the collection source, not the final tested material. A venous draw may produce a whole-blood result (Bld or BldV) or serum/plasma result (Ser/Plas). The validator therefore permits those blood-derived systems, while still rejecting arterial, capillary, urine, and cerebrospinal-fluid candidates. This is a general specimen rule, not a Wellspan-specific alias.

Ratio properties are also safety-critical. For example, '2325-9' is a GGT/AST ratio and cannot accept an observed 'IU/L' enzyme activity unit. A plain GGT row must use a standalone GGT term such as '2324-2', when the remaining evidence supports it; it must never be rescued by name similarity.

CBC Display Units And Axis Collisions

CBC reports commonly print K/mcL and M/mcL. These are display surfaces for 10^3/uL and 10^6/uL, respectively. The unit parser normalizes the mcL surface to uL and then applies the metric prefix, so the mapper compares dimensions rather than requiring a new alias for every laboratory's casing. FL is normalized to UCUM fL for platelet mean volume and RDW-SD.

One subtle catalog issue is that short analyte abbreviations can also occur as LOINC SYSTEM values for unrelated terms. For example, WBC is a valid system token in a small set of control terms, but it is not a specimen in a CBC row. The context extractor therefore does not infer a specimen when an unreviewed system token covers the entire analyte name. Explicit row specimen fields and reviewed specimen aliases remain authoritative. The same guard is applied while UMLS sibling labels are aggregated, so semantic enrichment cannot reintroduce the false specimen. UMLS fallback axis spans are also checked at token boundaries. This matters for short Part labels: Imm must not match the middle of immature, and Auto must not match the middle of automatic as a method clue. Without these guards, a correct CBC candidate can be rejected as if its specimen or method were invented by a sibling terminology label.

The Wellspan CBC registry entries are source-scoped and unit/property/class constrained. They are not global medical synonym rules. The active entries in config/mapping_registry.json include the report's automated differential rows and retain the source report and LOINC release in candidate evidence. A new laboratory does not inherit these source entries, but it still receives the universal Component/System/Time retrieval path before UMLS and SapBERT.

For the supplied Wellspan free-testosterone row, pg/mL leaves multiple release candidates possible, including calculated and detection-limit siblings. The source-scoped Testosterone, Free -> 2991-8 entry resolves that ambiguity only for the reviewed Wellspan surface and still runs UCUM, class, property, six-axis, ranking, and confidence validation. The method text LCMSMS is retained as provenance; it does not invent a LOINC method axis when the target term is intentionally methodless.

10A. Release-Derived Evidence

The compiled LOINC catalog imports COMMON_TEST_RANK and ORDER_OBS from the release. COMMON_TEST_RANK is a bounded retrieval prior: it helps common laboratory tests appear earlier, but it never filters rare tests and never overrides unit or axis safety. ORDER_OBS is a hard eligibility rule for a result row: only Observation and Both terms can be emitted. An order-only panel is not a result observation.

Recurring laboratory mappings belong in the governed source_mapping_evidence layer, not in a growing Python regex table. Each record keeps the local test identifier/name, laboratory, specimen, unit, result style, method, source document, reviewer, dates, release, proposed LOINC code, and review status. A proposed target remains inactive until a clinical reviewer confirms the exact vendor definition. A source record helps a matching Labcorp, Quest, Wellspan, or other report, but it cannot make universal retrieval unavailable for a new laboratory. A reviewer can promote a truly invariant display to universal_mapping_templates and retain the original source as supporting provenance.

For example, the pinned catalog can verify that 20448-7 is insulin in serum or plasma with property ACnc and UCUM example u[IU]/mL. It cannot determine from a blurry report whether the vendor's IU/mL surface has the same intended meaning; the source report and laboratory test directory must provide that evidence. The UCUM parser accepts both surfaces when they have the same dimension, but the clinical promotion decision remains review-gated.

10B. DSPy Judge Contract

The judge independently reloads the pinned LOINC catalog and runs the selected code through active-code, class, unit, specimen, and axis validation. Set LOINC_CATALOG_PATH (or NOISE_TO_LOINC_CATALOG) in the main project to the same catalog_v5.sqlite3 used by the mapper. If that artifact is unavailable, the judge fails closed rather than trusting an LLM suggestion.

The optional temp_file/judge.py adapter is a constrained review/UI layer; it is not a second LOINC authority. It receives the raw row, report context, normalization trace, routing, mapper status, mapper provenance, and complete safety-approved candidate objects. Candidate JSON is never truncated in the middle of a record.

Its structured output is:

{
  "decision": "ACCEPT_MAPPER | SUGGEST_REVIEW | ABSTAIN",
  "best_loinc": "a supplied candidate code or empty string",
  "confidence_band": "HIGH | MEDIUM | LOW",
  "reason": "one short sentence"
}

The adapter post-validates this output. ACCEPT_MAPPER is allowed only when the mapper already returned mapped, and its code is forced to the mapper's validated code. A non-mapped row can receive SUGGEST_REVIEW for a human screen, but its suggested code is never sent to Elation. If every candidate failed hard safety validation, the judge returns ABSTAIN and explains the actual count instead of incorrectly saying that no candidates existed.

This contract prevents an LLM from repairing a wrong unit, choosing an ionized/total sibling from text similarity, or inventing a LOINC code. The mapper remains the only component allowed to create an Elation test.loinc.

11. Elation Boundary

The mapper does not call Elation. ElationAdapter is a separate integration boundary:

payload = ElationAdapter().to_payload(observation, result)

If result.status is not mapped, this raises an error instead of emitting an unsafe or incomplete test.loinc value.

12. How to Run the Notebook

From the repository root:

$env:PYTHONPATH = "$PWD\src"
$env:HF_HUB_OFFLINE = "1"
python -m pip install jupyter
jupyter notebook notebooks/01_pipeline_walkthrough.ipynb

The notebook loads the local catalog and production assets. Catalog loading and SapBERT initialization may take time. It deliberately prints intermediate values instead of hiding them behind one mapper.map(...) call.

If the real assets are not available, read the markdown cells and use the small unit-test fixtures in tests/ to study deterministic behavior. The test doubles are for learning and tests only; production construction still requires UMLS/ScispaCy and SapBERT.

13. Using MIMIC Without Pretending It Is Gold

MIMIC-IV helps us learn what real hospital laboratory labels and units look like. The important mental model is:

MIMIC labevents + d_labitems
  -> local-label candidate corpus
  -> expert item-to-LOINC review
  -> approved mapping template
  -> evaluation gold CSV

labevents supplies observations such as values, units, and reference ranges. d_labitems supplies the local itemid, label, fluid, and category. The local label is not itself a LOINC code. In MIMIC-IV v3.1 the old LOINC field is not a trustworthy gold label, so the system intentionally starts every row as gold_status=unlabeled.

Run:

python -m loinc_mapper prepare-mimic `
  --labevents path/to/labevents.csv `
  --d-labitems path/to/d_labitems.csv `
  --output evaluation/mimic/candidates.csv `
  --mapping-template evaluation/mimic/item_mapping_template.csv

An expert then reviews the template using the same six-axis reasoning as the mapper: component, property, time, system, scale, method, and unit evidence. Only rows marked approved with an active LOINC code become evaluation rows:

python -m loinc_mapper build-mimic-gold `
  --candidates evaluation/mimic/candidates.csv `
  --mapping evaluation/mimic/item_mapping_template.reviewed.csv `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --output evaluation/mimic/gold.csv

Do not include MIMIC patient, encounter, specimen, timestamp, or provider identifiers in the exported corpus. Keep the source and derived files under the PhysioNet agreement. Split evaluation by itemid and local-label family, not by random lab rows, and keep a separate fax/OCR holdout for the final coverage and precision claim.

14. Proving the production path ran

The presence of a terms.faiss file is not enough. The saved MappingResult provenance is the runtime proof:

result.provenance["semantic_index"]
result.provenance["stages"]["semantic_retrieval_candidate_count"]
result.provenance["stages"]["semantic_stages_ran"]
result.provenance["umls_retrieval_backend"]

Expected values include a FAISS semantic index, the current UMLS backend, and semantic_stages_ran values of true for the required production stages. The semantic candidate count should normally be nonzero, but a zero count is not by itself proof that the stage was skipped. If the result says semantic_stages_ran["faiss"] == false or direct_encoder_rerank, the run did not use the catalog-wide FAISS retrieval stage. A zero hit count with the stage marked true is a retrieval-recall problem, not a stage-skipping signal. Production construction now fails when the pinned sidecar is missing rather than silently changing the flow.

The common spaCy warning [W036] The component 'matcher' does not have any patterns defined is not a mapping success or failure. It means the abbreviation component found no configured pattern for that short document; the local UMLS linker still receives the raw name and detected mentions. The mapping result and stage provenance, not warning absence, determine whether the pipeline ran.

Before using a stress CSV as gold, run the label audit:

python -m loinc_mapper audit-labels `
  --input evaluation/stress_test/stress_test_output_v2.csv `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --output evaluation/stress_test/stress_test_output_v2_label_audit.json

This catches contradictions such as a raw name saying Calculated while the expected LOINC method is empty, or a raw name saying serum while the expected LOINC system is whole blood. The safe behavior is to send those rows for clinical review, not to lower the safety threshold until the metric improves.

15. Batch Performance Without Shared Decisions

Mapper.map_many() is the production boundary for a report or CSV batch. It does not turn a report into one combined clinical statement. Instead, it:

  1. Runs ScispaCy/UMLS linking in grouped work where the adapter supports it.
  2. Reuses the loaded read-only SQLite catalog and FAISS index.
  3. Caches repeated SapBERT query embeddings and semantic hit lists.
  4. Calls the complete map() safety and confidence path for every observation.

The semantic stage uses at most two query hypotheses per row: the original OCR name and the first normalized/retrieval hypothesis. The deterministic stage still receives the complete bounded normalization hypothesis set. This keeps the mandatory SapBERT stage active while preventing every punctuation and OCR variant from causing another CPU embedding pass. The limit is recorded as stages.semantic_query_count in provenance.

map-batch, map-report, review-queue, and evaluate use this batch API. The output remains one result per input row, including rows that abstain.

16. The Performance Upgrade, in Plain Language

The required clinical logic did not become optional. We removed repeated work. Think of a 25-row fax as one folder given to one worker, not 25 workers each opening the same reference books.

old pattern:  row -> open/check UMLS -> query -> embed -> search -> close
new pattern:  report -> open once -> deduplicate -> batch query/embed -> map each row safely

What PipelineSettings controls

PipelineSettings reads .env and writes selected switches into each result's provenance.execution field. In normal production all evidence stages are on. This is not a speed-versus-safety knob.

PIPELINE_PROFILE=production
IS_UMLS_ON=true
IS_FAISS_ON=true
IS_SAPBERT_ON=true
IS_VALIDATION_ON=true
IS_CONFIDENCE_ON=true

Only PIPELINE_PROFILE=evaluation may turn a stage off to measure its impact. Those results are marked experimental, so they cannot become Elation payloads.

The exact registry fast path

The registry can save time only for a narrow case. It needs an exact, explicitly reviewed universal_template record:

{
  "alias": "VLDL Cholesterol Calculated",
  "target": "13458-5",
  "required_units": ["mg/dL"],
  "required_method": "Calculated",
  "review_status": "verified_active",
  "fast_path_enabled": true
}

Before skipping semantic work, the mapper checks the active LOINC code, the observed unit, extracted method/specimen, registry requirements, UCUM safety, all available LOINC axes, and confidence. A contained alias, legacy alias, source-specific evidence record, or missing required context does not fast-path. It uses the full pipeline.

Why a UMLS serving database exists

The raw UMLS database is an audit source. It is large because it contains broad terminology and an old explicit trigram table. The serving build creates a new database with only this question in mind:

Which UMLS names connect through a CUI to an active local LOINC code?

It has three ways to retrieve a name:

  1. Exact normalized lookup, for hemoglobin a1c.
  2. FTS5 token search, for words in a different order or with extra text.
  3. Trigram search followed by bounded rescoring, for OCR spelling noise.

The full checksum happens while building/deploying the artifact. At runtime the worker trusts its pinned manifest and file size, opens one read-only connection, and caches repeated names. That improves speed without weakening CUI-to-LOINC traceability.

Why FAISS gets two kinds of vectors

The regular FAISS index contains many vectors per LOINC code: long name, short name, consumer name, aliases, and Parts. That is good for finding a code. The code-level sidecar averages those alias vectors to one normalized vector per code. That is good for ranking a candidate we already found.

query embedding dot code vector = cosine similarity

Cosine similarity is one ranker feature. It never overrides a unit mismatch, wrong specimen, wrong method, or wrong LOINC property.

Why SapBERT loads locally

The worker first finds the pinned local SapBERT snapshot and loads it with local_files_only=True. This is deliberate: a fax worker must not depend on a network download or silently change model weights during a clinical mapping job. If Windows reports that the paging file is too small, the host needs more virtual-memory capacity; do not replace SapBERT with a smaller unvalidated model just to make that error disappear.

How to read timing evidence

Run one row, then inspect:

result.provenance["stages"]["timings_ms"]
result.provenance["stages"]["cache_hit"]
result.provenance["execution"]

Useful keys include deterministic_batch_ms, umls_and_context_enrichment_ms, faiss_retrieval_ms, validation_ms, ranking_ms, and total_ms. A cache hit marks cache_hit=true; it cannot be reused across a different artifact, registry, or pipeline-setting fingerprint.

Batch worker mental model

MappingBatch contains many reports, but each observation is still decided on its own. The worker flattens rows only to share catalogs, UMLS connections, and SapBERT batches. It restores report IDs around the results:

fax-1: [LDL, HbA1c]
fax-2: [SHBG]
        |
        v
one warm mapper.map_many([LDL, HbA1c, SHBG])
        |
        v
fax-1: [result, result]
fax-2: [result]

Candidate generation, validation, confidence, and provenance remain attached to one LabObservation at a time.

Clinician-Governed Learning Without a Black Box

When the mapper returns abstain, invalid_candidate, or no_candidate, the main fax project creates a review case. It stores the original OCR facts and mapper explanation unchanged, shows those facts to a staff member or clinician, and records any correction as a new audited revision. The mapper package does not store patient data or provide a UI.

The clinician sees a normal form rather than JSON:

Original report row -> corrected facts if the report proves OCR was wrong
                      -> choose active LOINC term
                      -> choose "all laboratories" or "only this laboratory"
                      -> write rationale -> approve

The package then validates the selected active LOINC code against the corrected facts. It runs the same UCUM, unit/property, specimen, method, time, scale, and six-axis checks as a normal mapping. There is deliberately no "force code" checkbox. A clinician may correct mgdL/ to mg/dL if the fax proves that fact, but cannot tell the package to accept an incompatible code.

An approved decision compiles to an immutable candidate registry snapshot. Normal clinician publication runs the reviewed rows together in one lightweight map_many() batch. It enables the proposed exact universal registry entry only inside that replay, checks active-code, context, UCUM, and six-axis safety, and then creates an active snapshot for newly launched mapper jobs.

policy = PublicationPolicy.CLINICIAN_FAST_REPLAY
snapshot = compile_registry_snapshot(active_registry, approvals, catalog, publication_policy=policy)
if snapshot.requires_publication:
    replay = replay_registry_snapshot(snapshot, approvals, review_mapper_factory, publication_policy=policy)
    active = finalize_registry_snapshot(snapshot, replay, publication_policy=policy)

review_mapper_factory uses PipelineSettings.clinician_review_replay(), so the publication batch does not load or invoke UMLS, ScispaCy, FAISS, SapBERT, or the learned ranker. The approval authority clears semantic confidence and margin only after exact alias, active-code, UCUM, and six-axis validation. It never bypasses a safety failure. Source-specific evidence remains on the full pipeline.

PublicationPolicy.STRICT_REPLAY retains the full original/unseen and fast-equivalence replays for offline release audits. It is intentionally not a browser-blocking prerequisite for an authorized clinician decision.

The current version does not inspect pending Elation orders. A future main project version may pass an advisory order-set/test identifier, local test ID, CPT, diagnosis, and ordering context after demographic matching. Those facts must never become a hard global LOINC filter.

The larger FastAPI project owns the review UI, clinician authentication, durable audit ledger, CSV upload, registry storage, and Cloud Run orchestration. This module supplies build_review_case, validate_review_decision, compile_registry_snapshot, replay_registry_snapshot, and finalize_registry_snapshot. See Clinician-governed learning for the complete field list and integration sequence.