Skip to content

Architecture

OCR text
  -> RowExtractor
  -> LabObservation
  -> lossless normalizer + generated hypotheses
  -> AxisFactExtractor (component/system/time/property/scale/method facts)
  -> UniversalSemanticIndex (release-derived source-neutral candidates)
  -> deterministic registry/LOINC retrieval
  -> LOINC Part and axis retrieval
  -> ScispaCy mention + abbreviation processing
  -> local UMLS exact/FTS/trigram retrieval
  -> candidate union
  -> hard UCUM/six-axis safety gate
  -> FAISS/SapBERT vector retrieval + validated reranking
  -> feature ranker + confidence calibration
  -> confidence policy
  -> MappingResult
  -> ElationAdapter

The production public boundaries are ReportMapper.map_report(report_text) and Mapper.map(LabObservation). Every production construction requires a configured UMLS/ScispaCy linker and SapBERT reranker; there is no unvalidated runtime bypass. The result is deliberately explainable and contains candidate evidence, axis comparisons, unit decisions, release metadata, and review reasons.

The one deliberate exception is the verified exact universal-template fast path. It is not a looser registry shortcut: it is available only to a verified_active, exact universal_template alias whose configured target, unit, class, property, specimen, method, context, active-code status, UCUM validation, six-axis validation, and confidence policy all pass. It records the skipped semantic stages in provenance. Contained aliases, legacy registry entries, and all source_mapping_evidence entries always use the full flow. An exact source test identifier may later disambiguate a documented local assay, but a laboratory name alone can never displace a verified universal template.

Before candidate retrieval, the mapper resolves the observation domain and optional LOINC laboratory class. LAB_RESULT is a hard scope and remains limited to laboratory class type 1. LOINC CLASS, panel context, and OCR section labels are advisory retrieval/ranking evidence, never clinical vetoes: a mixed report section cannot erase a valid CRP, endocrine, or chemistry term before its six axes and unit are checked. Imaging, documents, and history are separate future adapters. See Universal multi-lab mapping and Category-aware routing.

AxisFactExtractor turns row facts such as Creatinine, Urine, 24 hour, CSF, Calculated, and an explicit method field into traceable hypotheses. UniversalSemanticIndex is built once from the loaded pinned catalog and uses active result terms, Components, Parts, consumer/related names, linguistic variants, and catalog token retrieval to create source-neutral candidates. It does not create a second terminology authority or rely on a source laboratory map. AxisSignature groups safe candidates by Component, Property, Time, System, Scale, and Method before confidence scoring. Explicit facts eliminate incompatible siblings; indistinguishable clinically distinct signatures still abstain.

Deterministic, Part, UMLS, and semantic retrieval are independent candidate sources. They are deduplicated by LOINC code while retaining all source evidence. The hard validator runs before learned ranking, so a high semantic score can never rescue an incompatible unit, specimen, method, time, or scale.

The release compiler creates assets/loinc/2.82/catalog_v5.sqlite3 with an indexed alias_tokens table, SQLite FTS5 prefix fallback, token document frequencies, and generated trigram sidecars. The normal retrieval path consults the token table first; FTS5 is not asked to group every alias containing a common word. Character retrieval uses distributed trigram intersections so a single OCR edit remains recoverable without a broad OR scan. The current catalog schema is version 6; it preserves release-derived COMMON_TEST_RANK and ORDER_OBS fields with indexes. COMMON_TEST_RANK is only a small candidate-ordering prior; it never excludes rare terms or overrides safety. ORDER_OBS is enforced before ranking so result rows can only emit Observation or Both terms. aliases_code_idx prevents candidate explanation from scanning the four-million-row alias table. Builds write a sibling temporary database and atomically replace the release artifact only after completion, so an interrupted build cannot expose malformed SQLite. Schema 6 also stores the generated axis vocabulary in axis_values, axis_aliases, axis_fts, and axis_trigram. Runtime context extraction uses these release-derived values plus versioned reviewed surface forms. Older artifacts use a catalog-term fallback and report that backend explicitly. The raw licensed UMLS release remains immutable. build-umls-serving derives serving/umls_loinc.sqlite3, retaining only aliases whose CUIs bridge to active LOINC codes and indexing them with exact lookup, FTS5, and trigram retrieval. Its runtime backend should report serving_exact+fts5+trigram; a raw fallback is supported only for transition and should be replaced before production latency measurements. The SapBERT compiler creates terms.faiss plus metadata containing the model, release, dimension, and record provenance. A code-vector sidecar prevents candidate embeddings from being recomputed. FAISS IndexFlatIP is the production backend; the exact NumPy backend exists only for tests/development.

The mapper uses one loaded catalog/vector index per worker and bounded in-process caches. map_many() is the shared execution engine for reports, CSVs, and the batch worker: it deduplicates deterministic retrieval, ScispaCy/UMLS mentions, and SapBERT embeddings while keeping each row's validation and confidence decision independent. Optional FAISS laboratory/class shards are derived from the global index and fall back to laboratory/global retrieval; they never become separate terminology databases. Redis and ClickHouse are intentionally not first-line dependencies for this approximately 97,000-term terminology catalog.

The vector sidecar is a required production retrieval stage, not a safety authority. It supplies catalog-wide SapBERT candidates and precomputed query similarities. The mapper still reranks validated non-sidecar candidates with SapBERT when needed, but production startup fails if the pinned sidecar is missing. This prevents an evaluation from silently measuring a different architecture than the deployed system.

Every result records semantic_index, semantic query/hit counts, UMLS backend, candidate source counts, hard rejections, cache state, profile, artifact versions, stages.semantic_stages_ran, and stages.timings_ms. The boolean stage map records that a stage executed even when it produced zero concepts or zero hits; candidate counts are evidence volume, not execution flags. A result with semantic_stages_ran["faiss"] == false or ranker backend direct_encoder_rerank did not execute the production FAISS retrieval path; zero hits with the stage marked true is a recall issue to investigate.

For a UMLS concept that has multiple sibling CUIs, the mapper retains direct CUI-to-LOINC bridge codes and performs a bounded local exact/token expansion of each concept label. It deliberately does not replay OCR normalization, registry matching, broad fuzzy retrieval, and context FTS for every sibling. That reduces duplicate work while preserving the candidate union and the later six-axis/UCUM gate.

Reviewed registry mappings and reviewed axis aliases are the only automatic vocabulary expansion. The registry has two governed layers: universal_mapping_templates for clinically invariant, expert-confirmed surfaces and source_mapping_evidence for a particular lab/test ID/method. Source evidence participates only for an exact source-laboratory match and can never suppress universal LOINC retrieval. A matching source_test_id may disambiguate a documented clinically distinct local assay; source-laboratory display evidence alone cannot. An approved universal template may retain a supporting source in provenance without requiring it at runtime. An uncertain result is exported to the review queue; an explicit expert approval can update terminology data without retraining SapBERT. See Universal multi-lab mapping.

Clinician-published entries carry authority=clinician_override, but that authority only clears confidence and margin reasons after exact-alias, active-code, UCUM, and six-axis validation. Normal clinician publication runs one lightweight exact map_many() replay with semantic stages disabled, then activates the verified universal fast path. Strict full/unseen plus fast-path equivalence remains a separate release-audit mode. Source-specific entries never use the fast path. Replay diagnostics expose bounded ranked candidate display names and registry evidence without report or patient payloads.

Clinician publication also stores bounded verified_aliases generated from the approved alias using normalized/compact and reviewed OCR-confusion surfaces. Exact matches to those replay-certified surfaces can carry the same authority; contained and arbitrary fuzzy matches remain ordinary retrieval evidence and must satisfy the normal confidence gate.

When a source-specific legacy row and a universal clinician row have the same alias and target, both remain represented for provenance, but the candidate merge chooses the clinician universal evidence. For different targets, an exact source test identifier is allowed to outrank universal evidence; a source-laboratory name alone is not. A same-scope legacy row is upgraded during snapshot compilation rather than duplicated.

Blood, Venous is interpreted as venous collection-source evidence. It may lead to whole-blood (Bld/BldV) or processed serum/plasma (Ser/Plas) terms; the analyte, units, and remaining axes still decide the final code. The UCUM parser handles composable forms such as u[IU]/mL, rather than maintaining an insulin-specific unit alias.

The public model keeps collection_specimen separate from specimen so collection context is not mistaken for analytical material. The report adapter preserves duplicate row positions rather than deduplicating by analyte name. This prevents a second Calcium row from disappearing before retrieval. Year-specific method terms, including CKD-EPI 2021, require the year in the row or a reviewed source mapping. Ratio properties such as CRto are hard incompatible with dimensional enzyme units such as IU/L.

The same grammar handles CBC display forms such as K/mcL, M/mcL, and FL. They are normalized to 10^3/uL, 10^6/uL, and fL before dimension checks. The context extractor also guards against catalog SYSTEM tokens that collide with an analyte abbreviation, such as WBC; a release token covering the entire analyte is not promoted to an inferred specimen unless it is a reviewed specimen surface or explicitly supplied by the report parser. This filter is also applied while aggregating UMLS sibling context so semantic enrichment cannot reintroduce a false specimen after raw-name extraction removed it. All UMLS fallback axis spans also require token-boundary presence in the raw label; this prevents short Part tokens such as Imm or Auto from matching inside immature or automatic and inventing a method.

Wellspan CBC and free-testosterone mappings are source-laboratory evidence in the active registry, constrained by class, unit, property, and release. This is deliberate: Testosterone, Free has multiple valid LOINC siblings in pg/mL, while the CBC abbreviations need report-specific interpretation of the automated differential. They can improve recurring-report coverage, but the ordinary source-neutral path still runs for all unseen laboratories.

The Elation adapter is separate from the mapper so the package can be embedded in another fax automation service without coupling the ranking engine to an EHR client.

For operational learning, the package exposes a storage-free clinician review contract. The main project owns authenticated UI, review persistence, immutable registry-object publication, and Cloud Run Job orchestration; this package validates decisions, compiles candidate snapshots, and replays them before an active snapshot is allowed. See Clinician-governed learning.