Skip to content

Deployment and Main-Project Integration

This page is the handoff contract for integrating the mapper into the larger fax-automation service. The mapper is a terminology and safety service; it is not an OCR engine, fax transport, patient-matching service, or Elation client. Those responsibilities remain in the main project.

Production Boundary

fax/PDF
  -> main project's OCR and row extraction
  -> LabObservation adapter
  -> one long-lived Mapper per worker
  -> MappingResult
  -> accepted-only ElationAdapter payload
  -> Elation API

The mapper must receive the complete row context whenever it is available: name, value, unit, specimen, collection specimen, method, time context, scale, reference range, source laboratory, observation domain, LOINC class hint, panel context, and report context. Do not reduce a row to only its name if the OCR system extracted a unit or specimen. Unit and axis evidence are safety inputs, not optional decoration.

source_laboratory is optional enrichment, not a required routing key. An ordinary result must map through the release-derived universal axis path when the laboratory is absent or appears for the first time. Use collection_specimen for a draw/source (for example Blood, Venous) and use specimen only when the report identifies the analytical material. LOINC class and panel context are advisory retrieval/ranking hints; LAB_RESULT is the hard boundary that limits output to laboratory result terms.

For a typed integration, send the canonical specimen enums in addition to the raw OCR text:

LabObservation(
    raw_name="Hemoglobin",
    value="12.0",
    unit="g/dL",
    specimen="blood",
    specimen_code="blood_unspecified",
    collection_specimen="Blood, Venous",
    collection_specimen_code="blood_venous",
)

Analytical values are unknown, blood_unspecified, whole_blood, serum, plasma, serum_or_plasma, urine, cerebrospinal_fluid, saliva, stool, sputum, amniotic_fluid, synovial_fluid, pleural_fluid, peritoneal_fluid, body_fluid_unspecified, tissue, and other. Collection values are unknown, blood_venous, blood_arterial, blood_capillary, urine_clean_catch, urine_catheter, urine_24_hour, and other. Preserve the original words for audit. Bld is a LOINC System-axis value, not a UI specimen enum. Contradictory text and enum values are rejected.

Artifact Bundle

Build or provision one matched artifact bundle. Do not mix releases between these files. A worker needs the first group; the raw UMLS files are retained only in the protected build/audit location after the serving index is built:

assets/loinc/2.82/catalog_v5.sqlite3
assets/loinc/2.82/catalog_v5.manifest.json
assets/loinc/2.82/terms.faiss
assets/loinc/2.82/terms.json
assets/loinc/2.82/terms.code_vectors.npy
assets/loinc/2.82/terms.code_vector_codes.json
assets/loinc/2.82/shards/vector_shards.json       # optional, benchmarked only
assets/umls/2026AA/serving/umls_loinc.sqlite3
assets/umls/2026AA/serving/serving_manifest.json
config/mapping_registry.json
config/axis_context_aliases.json
config/panel_context_rules.json
config/ocr_corrections.json
config/unit_surface_rules.json

assets/umls/2026AA/umls.sqlite3                   # protected rebuild/audit only
assets/umls/2026AA/manifest.json                  # protected rebuild/audit only

The LOINC CSV files under data/ are immutable build inputs. Runtime mapping uses the compiled schema-6 SQLite catalog, not a CSV scan. The catalog imports release-derived COMMON_TEST_RANK and ORDER_OBS; the former is a bounded ranking prior and the latter prevents order-only terms from representing a result row. The FAISS file is derived from the same LOINC release and SapBERT model recorded in terms.json. The UMLS database is licensed material and must be provisioned through a protected artifact store or deployment secret volume, never committed to Git.

UniversalSemanticIndex is built in memory once from this pinned SQLite catalog when the worker constructs a Mapper; it has no separate artifact and does not duplicate LOINC authority. It uses Components, Parts, release aliases, consumer/related names, and axis facts to reach ordinary source-neutral candidates before UMLS/FAISS recall.

Workers prefer serving/umls_loinc.sqlite3 when present. It is the compact runtime artifact; retain raw UMLS only in protected rebuild/audit storage. Code-vector files avoid repeated candidate embedding during ranking. Class shards are optional latency artifacts, never another terminology authority.

Preflight

Run these checks in the image build or release pipeline, not for every row:

$env:PYTHONPATH = "$PWD\src"
python -m loinc_mapper validate-assets `
  --umls-path assets/umls/2026AA
python -m unittest discover -s tests -v
python -m mkdocs build --strict

The UMLS checksum check reads the entire raw SQLite file and can take minutes for the approximately 14 GB artifact. That is expected. Run it in the image build/release validation job, not worker startup. A worker validates the pinned serving manifest and file size instead of hashing raw UMLS data.

Perform one real smoke mapping and inspect its provenance:

python -m loinc_mapper map `
  --name "Cholesterol, LDL, Measured" `
  --value "133" `
  --unit "mg/dL" `
  --umls-path assets/umls/2026AA `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --vector-index assets/loinc/2.82/terms.faiss `
  --vector-metadata assets/loinc/2.82/terms.json `
  --scispacy-model en_core_sci_md

The smoke result must show semantic_index.backend=faiss, a nonzero semantic_retrieval_candidate_count, axis_vocabulary.backend=sqlite, and umls_retrieval_backend=serving_exact+fts5+trigram. It must not report a NumPy vector backend or exact_only_bridge.

For the universal layer, also inspect result.provenance["stages"]["universal_axis_facts"] and result.provenance["stages"]["axis_signature_groups"]. A normal urine, CSF, 24-hour, calculated, or direct-assay row should show the report facts that removed its incompatible LOINC siblings.

result.provenance["stages"]["registry_evidence"] shows separately whether a universal template or source-specific record participated. routing_behavior shows that class metadata was advisory and laboratory-wide retrieval remained available; it is useful when an OCR section label is wrong.

Registry Approval Replay

Clinician approvals are published as authority=clinician_override, but this authority clears only semantic confidence and margin after normal UCUM, active-code, six-axis, context, and exact-alias checks pass. It is not a force map. The main project uses the fast publication policy for clinician-approved universal templates:

policy = PublicationPolicy.CLINICIAN_FAST_REPLAY
candidate = compile_registry_snapshot(
    active_registry,
    approvals,
    catalog,
    publication_policy=policy,
)
if candidate.requires_publication:
    # Factory builds with PipelineSettings.clinician_review_replay().
    replay = replay_registry_snapshot(
        candidate,
        approvals,
        mapper_factory,
        publication_policy=policy,
        progress=record_progress,
    )
    active = finalize_registry_snapshot(
        candidate,
        replay,
        publication_policy=policy,
        progress=record_progress,
    )

This performs one deduplicated map_many() batch over the original reviewed rows. The review mapper loads no UMLS, ScispaCy, FAISS, SapBERT, or learned ranker assets. It still performs deterministic context, exact alias, active term, UCUM, and six-axis checks. A rejected unit/specimen/method/time/scale cannot publish just because a clinician selected a code.

PublicationPolicy.STRICT_REPLAY remains the scheduled release-audit mode. It performs full original/unseen and fast equivalence replays, but it must not be used synchronously from a clinician browser action.

Publication generates bounded verified_aliases from the approved name for lossless/compact and reviewed OCR variants. These variants remain linked to the approval and are safe only after replay and normal validation. A contained or arbitrary fuzzy registry hit does not inherit clinician authority.

If the active registry already contains legacy source evidence for the same alias and target, the publisher should not delete it. The compiled snapshot adds the approved universal template, and runtime candidate merging prefers the clinician-approved universal evidence for that code. If the legacy entry has the same scope as the approval, compilation upgrades that entry in place.

Workers must load only a finalized registry_status=active snapshot. Candidate snapshots and browser-submitted registry JSON must be rejected. The active pointer should be advanced with a Cloud Storage generation precondition so two publishers cannot overwrite one another.

Asynchronous Publisher Job

The FastAPI request persists the append-only decision and starts a private registry-publisher Cloud Run Job. It returns the review status immediately; the browser polls the main project rather than waiting on mapper work. The job:

  1. Loads the active immutable snapshot and only pending/superseding decisions.
  2. Emits validating, batch_queued, batch_completed, and finalization_ready progress from PublicationProgressEvent.
  3. Skips replay and registry writes for candidate.requires_publication=false, recording no_change for those audit decisions.
  4. Writes the finalized immutable snapshot, advances the active pointer with a generation precondition, then records published.
  5. Records failed_safety with bounded de-identified diagnostics when deterministic validation or the exact replay fails.

Keep strict model replays in a separate scheduled Cloud Run audit job. Those audits may create clinician-visible follow-up work but never silently roll back an already authorized clinician approval.

Deferred Order Context

Pending Elation orders are intentionally deferred to the next version. After demographic matching, the main project may inspect the patient chart and pass an advisory OrderContext with an order-set/test identifier, local lab test identifier, CPT, diagnosis, and ordering context. It must not be a hard global LOINC filter. If it is absent, stale, ambiguous, or unsafe, the mapper uses normal laboratory-wide retrieval.

The current practice CPT CSV is advisory inventory only, not an active practice-specific subset, so it must not filter the LOINC catalog. Pasted order sets follow the same rule until the future order-context integration is clinically mapped and replay-tested.

SapBERT is loaded from the pinned local model snapshot only; production workers do not download it from Hugging Face at request time. A Windows error such as The paging file is too small for this operation means the host virtual-memory configuration cannot reserve the model weights. Increase the paging file or run the worker on a host/container with sufficient memory; do not replace SapBERT with a smaller unvalidated model as a workaround.

Python Integration

Instantiate the mapper once during worker startup. Do not construct ScispaCy, UMLS connections, SapBERT, or FAISS inside a request loop.

from pathlib import Path

from loinc_mapper import LabObservation, build_mapper
from loinc_mapper.elation import ElationAdapter

ROOT = Path("/opt/noise-to-loinc")
VECTOR_SHARDS = ROOT / "assets/loinc/2.82/shards/vector_shards.json"
mapper = build_mapper(
    core_path=ROOT / "data/Loinc_2.82/Loinc_2.82/LoincTableCore/LoincTableCore.csv",
    rich_path=ROOT / "data/Loinc_2.82/Loinc_2.82/LoincTable/Loinc.csv",
    common_names_path=ROOT / "data/loinc_common.csv",
    registry_path=ROOT / "config/mapping_registry.json",
    map_to_path=ROOT / "data/Loinc_2.82/Loinc_2.82/LoincTable/MapTo.csv",
    catalog_path=ROOT / "assets/loinc/2.82/catalog_v5.sqlite3",
    umls_path=ROOT / "assets/umls/2026AA",
    scispacy_model="en_core_sci_md",
    sapbert_model="cambridgeltl/SapBERT-from-PubMedBERT-fulltext",
    vector_index_path=ROOT / "assets/loinc/2.82/terms.faiss",
    vector_metadata_path=ROOT / "assets/loinc/2.82/terms.json",
    vector_shards_path=VECTOR_SHARDS if VECTOR_SHARDS.exists() else None,
)

observation = LabObservation(
    raw_name=ocr_row.name,
    value=ocr_row.value,
    unit=ocr_row.unit,
    specimen=ocr_row.specimen,
    collection_specimen=ocr_row.collection_specimen,
    method=ocr_row.method,
    time_context=ocr_row.time_context,
    scale=ocr_row.scale,
    reference_range=ocr_row.reference_range,
    source_laboratory=ocr_row.source_laboratory,
    observation_domain=ocr_row.observation_domain or "LAB_RESULT",
    loinc_class_hint=ocr_row.loinc_class_hint,
    panel_context=ocr_row.panel_context,
    class_source=ocr_row.class_source,
    class_confidence=ocr_row.class_confidence,
    report_context=ocr_row.report_context,
)
result = mapper.map(observation)

if result.status == "mapped":
    payload = ElationAdapter().to_payload(observation, result)
    # Send payload to Elation only after recording result.provenance.
else:
    # Store result.to_dict() in the review queue. Never guess a LOINC code.
    payload = None

When diagnosing an unexpected abstention, verify the worker loaded this workspace's package and registry rather than an older installed copy:

import inspect
import loinc_mapper

print("loinc_mapper:", inspect.getfile(loinc_mapper))
print("registry:", mapper.registry.version)
probe = mapper.map(LabObservation(
    raw_name="SEX HORMONE BINDING GLOBULIN",
    value="162.00",
    unit="nmol/L",
    specimen="Blood",
    observation_domain="LAB_RESULT",
    panel_context="TESTOSTERONE, TOTAL AND FREE",
))
print(probe.status, probe.loinc_code)
print(probe.provenance.get("mapping_registry_version"))

With the reviewed endocrine aliases, the probe should return mapped and 13967-5. A worker that reports an older registry version, imports the main project's legacy faxautomation lookup, or returns abstain has not loaded the current mapper artifact. Restart that worker after updating the package and config/mapping_registry.json.

For a fax with several rows, use mapper.map_many(observations). This reuses the release-derived universal index, UMLS lookups, FAISS query embeddings, SQLite access, and process memory while keeping every row's safety decision independent.

Close the mapper when a short-lived job exits so Windows and container runtime file handles are released cleanly:

try:
    results = mapper.map_many(observations)
finally:
    mapper.close()

If the main project already owns OCR table extraction, use its parser and pass rows through the adapter above. Use map-report only when the input is a text report and the package's replaceable report extractor is appropriate.

Result Contract

Persist the complete MappingResult.to_dict() for audit and review. At a minimum, retain:

  • status, loinc_code, confidence, and margin.
  • normalized input and normalization operations.
  • candidate codes and retrieval sources.
  • six-axis and unit evidence, including hard rejections.
  • UMLS, FAISS, model, catalog, registry, and release provenance.
  • abstention or rejection reasons.

Only status == "mapped" may create an Elation test.loinc value. Treat abstain, no_candidate, and invalid_candidate as review work, not as failed API calls to retry blindly.

If the main project uses the DSPy review adapter in temp_file/judge.py, pass the complete MappingResult.to_dict() and the original row context. The judge may explain or suggest a safety-approved candidate for a reviewer, but the main project must still use the mapper status/code for filing. Never use loinc_judge_code as an Elation code when the mapper status is not mapped. Persist the judge decision and reason beside the mapper provenance for audit. Configure LOINC_CATALOG_PATH in the main project to the same pinned assets/loinc/2.82/catalog_v5.sqlite3. The judge independently checks active status, class, UCUM dimension, specimen, and all six axes. It fails closed if the catalog is missing. Do not let a judge code become an Elation code.

The OCR adapter must preserve duplicate analyte rows. A dictionary keyed only by name will lose a second Calcium, bilirubin, or hormone row. Use a list of row objects with page/report identity and pass one LabObservation per row. Pass collection-only text via collection_specimen; do not claim that a venous collection proves the analytical material is serum/plasma or whole blood.

Clinician Review, Registry Learning, and Publishing

The production review path is described in detail in Clinician-governed learning. The main project must persist review cases, case revisions, mapper provenance, clinician decisions, reviewer identity, status history, report references, and registry-version references in a durable ledger. This package deliberately has no frontend, HTTP routes, Firestore client, Cloud Storage client, or patient-data database.

Staff can draft or correct facts, while an authorized clinician approves a reusable mapping with rationale. The UI must show the original OCR row, report/page link, mapper reason, safe candidates, active LOINC details, six axes, and example units. Do not expose raw registry JSON, a force-map option, or fast_path_enabled to clinicians.

The application boundary is:

review_case = build_review_case(observation, result, report_reference)
validation = validate_review_decision(review_case, decision, catalog, active_registry)
policy = PublicationPolicy.CLINICIAN_FAST_REPLAY
candidate = compile_registry_snapshot(active_registry, approvals, catalog, publication_policy=policy)
if candidate.requires_publication:
    replay = replay_registry_snapshot(candidate, approvals, mapper_factory, publication_policy=policy)
    active_snapshot = finalize_registry_snapshot(candidate, replay, publication_policy=policy)

correct_and_recheck creates an audited correction and remaps the row. It does not create a registry mapping. not_standardized and not_lab_result are manual outcomes that never emit test.loinc. The clinician-fast policy runs one original-row map_many() replay using PipelineSettings.clinician_review_replay() and immediately enables the exact universal fast path after it passes. Source-specific entries remain full-pipeline only. Clinician authority clears confidence and margin only after exact-alias, active-code, UCUM, and six-axis validation; it is not a force-map.

Use PublicationPolicy.STRICT_REPLAY only in an asynchronous audit/release job. It retains original/unseen full-pipeline replay and fast equivalence checks without blocking clinician publication.

The candidate snapshot has registry_status=candidate, so mapper workers reject it. Only the finalized replay-passing payload is active and can be published as a new immutable registry object.

Legacy JSONL Diagnostics

The following compatibility commands are for local diagnostics or migration of an old JSONL queue. New clinician approvals should use the review-case contract and CSV workflow instead. The compatibility importer never enables the registry fast path.

python -m loinc_mapper review-queue `
  --input observations.csv `
  --output evaluation/review_queue.jsonl `
  --umls-path assets/umls/2026AA

python -m loinc_mapper import-reviews `
  --input evaluation/review_queue.approved.jsonl `
  --output config/mapping_registry.reviewed.json

Review data must identify the reviewer, decision, selected active LOINC code, and rationale for unit, specimen, method, and other axis choices. Registry updates are vocabulary changes, not automatic model retraining. Promote them through normal review and deployment controls.

When an expert approves a review row, set review.mapping_scope explicitly:

  • universal_template only when the display means the same clinical measurement across laboratories and its unit/axis constraints are known.
  • source_evidence for a local identifier, vendor assay, or report-specific display that needs the named source laboratory.

For a universal approval sourced from one laboratory, the importer records that laboratory as supporting_source_laboratory in provenance rather than making it an execution-time requirement. Audit and export the two layers without changing the live registry:

python -m loinc_mapper audit-registry-scopes `
  --registry config/mapping_registry.json `
  --output evaluation/registry_scope_audit.json

python -m loinc_mapper export-layered-registry `
  --registry config/mapping_registry.json `
  --output config/mapping_registry.layered.json

Cluster recurring abstentions before requesting clinical review:

python -m loinc_mapper cluster-review-queue `
  --input evaluation/review_queue.jsonl `
  --output evaluation/review_clusters.json

Operational Safeguards

  • Load one immutable artifact bundle per worker and expose its versions in health metadata.
  • Do not send an unmapped or abstained result to Elation with a guessed code.
  • Keep UMLS files and patient data out of logs, Git, container layers, and error reports.
  • Set a resource limit for concurrent mappings; SapBERT is CPU/GPU and memory intensive.
  • Use batch mapping for reports and avoid parallel workers competing for the same large SQLite file during artifact builds.
  • Alert on increased abstention rate, dangerous unit/property rejections, missing FAISS evidence, or a UMLS backend downgrade.
  • Retain enough provenance to reproduce a decision after a release update.

Batch Worker and Cloud Run Job

Use MappingBatch instead of starting a mapper for every fax. Group incoming reports for up to 60 seconds or 20 reports / 500 observations, then submit one Cloud Run Job task. The task creates one mapper process, maps the whole batch, persists the result, and exits.

{
  "schema_version": "1",
  "batch_id": "fax-batch-001",
  "registry_snapshot": {
    "registry_version": "registry-20260807T0100-abc123",
    "uri": "gs://private-bucket/registries/mapping_registry.registry-20260807T0100-abc123.json",
    "generation": "1722980000000000",
    "sha256": "..."
  },
  "reports": [
    {
      "report_id": "fax-123",
      "observations": [{"raw_name": "Hb A1c", "value": "6.5", "unit": "%"}]
    }
  ]
}

python -m loinc_mapper run-worker `
  --input batch.json `
  --output batch-result.json `
  --umls-path assets/umls/2026AA

Start Cloud Run Jobs with one task at a time, 4 vCPU, 8 GiB, and scale-to-zero. Set MAPPER_CPU_THREADS=4 so Torch, FAISS, and BLAS do not oversubscribe the worker. Increase memory only from measured peak RSS. The main fax service owns callback delivery, retries, idempotency, and durable mapping/review status; persist the mapper result before Elation submission.

At job start, the launcher downloads the named active registry snapshot to ephemeral local storage, verifies its checksum, constructs Mapper from that file, and passes the same reference in MappingBatch. BatchWorker refuses to run if the loaded registry version differs from the batch reference. This lets already-running jobs finish against their explicit old snapshot while new jobs use a newly approved one.

Keep PDFs, OCR input/output, mapper results, and registry snapshots in private Cloud Storage. Keep raw licensed UMLS build data in a protected build/audit location; package only the compact serving artifact and pinned model/catalog bundle in the private worker image. Do not bake raw licensed data or mutable registry content into a public image.

Use separate least-privilege service accounts:

  • Main API: read/write review ledger and report storage; trigger Jobs.
  • Mapper Job: read its batch and finalized registry snapshot; write mapping results; call an authenticated internal completion route.
  • Registry publisher: read approved review events; write versioned snapshots and atomically update the active pointer.

Use immutable Cloud Storage object names and a generation precondition when updating the active pointer. See Cloud Storage request preconditions and Cloud Run Jobs. If using Firestore for the review ledger, deployment still requires the organization's executed BAA, IAM controls, audit logging, retention policy, and security review; see Google Cloud HIPAA covered products.

Serving-Artifact Build

Build the UMLS serving index in a separate protected build job. It validates the raw release checksum once and can take substantial temporary disk space; it does not modify the raw UMLS index. On Windows, launch it in the background and follow the progress log on standard error:

$env:PYTHONPATH = "$PWD\src"
$env:PYTHONUNBUFFERED = "1"

New-Item -ItemType Directory -Force evaluation\logs | Out-Null

$job = Start-Process `
  -FilePath "$PWD\.venv\Scripts\python.exe" `
  -ArgumentList @(
    "-u",
    "-m", "loinc_mapper",
    "build-umls-serving",
    "--asset-path", "assets\umls\2026AA",
    "--catalog", "assets\loinc\2.82\catalog_v5.sqlite3",
    "--output", "assets\umls\2026AA\serving\umls_loinc.sqlite3"
  ) `
  -WorkingDirectory "$PWD" `
  -RedirectStandardOutput "evaluation\logs\umls_serving_build.out.log" `
  -RedirectStandardError "evaluation\logs\umls_serving_build.err.log" `
  -WindowStyle Hidden `
  -PassThru

$job.Id
Get-Content evaluation\logs\umls_serving_build.err.log -Wait

After it exits, run validate-assets once in the build/deployment pipeline and confirm a smoke mapping reports serving_exact+fts5+trigram. Do not deploy the new artifact until the profile matrix and safety regression checks pass.

Pipeline Profiles

PIPELINE_PROFILE=production requires all evidence stages. It permits an early result only for a fully validated verified_active exact registry entry. PIPELINE_PROFILE=evaluation permits controlled ablations but marks results experimental, and ElationAdapter rejects those results. Use the profile matrix on a frozen expert-reviewed corpus before changing a production stage or the registry fast-path flag.

Deployment Promotion

Use a new versioned artifact directory for every LOINC or UMLS release. Run tests, asset validation, smoke mappings, and the expert holdout evaluation before changing the worker's active configuration. Promote by changing the artifact pointer/configuration and restarting workers. Roll back by restoring the previous pointer; do not edit a live SQLite file or delete the previous bundle while workers may still have it open.