Skip to content

Clinician-Governed Learning

This module provides the validation contract for clinical mapping review. It does not provide a web UI, a database, patient-data persistence, Firestore, Cloud Storage, or HTTP endpoints. The main fax-automation project owns those responsibilities so this package remains portable and stateless.

mapper result
  -> main project stores a review case
  -> staff drafts corrections or a clinician approves a decision
  -> module validates the decision against the pinned LOINC release and safety rules
  -> main project compiles, replays, and publishes an immutable registry snapshot
  -> the next mapping job loads that exact snapshot

People and Responsibilities

Role Main responsibility
Staff reviewer Creates a draft, corrects obvious OCR facts, and attaches the report/page. Cannot activate a reusable mapping.
Authorized clinician Selects a LOINC code, chooses the scope, records rationale, and approves a reusable decision.
Main project Authenticates users, stores review history, serves the clinician UI, uploads CSV files, and starts mapping jobs.
Registry publisher Validates approvals, runs replay, writes immutable snapshots, and atomically changes the active pointer.
noise_to_loinc Searches active LOINC terms, validates decisions, compiles registry content, and replays mappings.

The UI must show a clinician the original OCR row, report/page link, mapper status and reason, ranked candidates, official LOINC name, six axes, example units, and active status. The clinician should never need to edit JSON.

What Happens to Each Outcome

Mapper result or review outcome Main-project action
mapped Use the existing accepted-only Elation path. Store provenance for audit.
abstain Show safe candidates and request a clinician choice or missing context.
invalid_candidate Show the hard safety reason. Let a reviewer correct OCR facts such as unit, analytical specimen, method, time, or scale, then remap.
no_candidate Let the clinician search active LOINC terms, select one, and provide supporting facts.
not_standardized review decision Keep the document for manual handling. Do not emit test.loinc.

An approved observation should map in the next newly launched job when the same clinical facts recur. A new row with a different or conflicting unit, specimen, method, time, or scale can still correctly abstain. The system must not promise that every future unknown clinical meaning will map automatically.

Doctor-Friendly Review Form

The main project should present this sequence, in plain language:

  1. Confirm the original report row and page.
  2. Correct an OCR fact only when the source document supports it: name, unit, analytical specimen, collection specimen, method, time, or scale.
  3. Search active LOINC terms by code, official name, or synonym.
  4. Inspect the chosen code's component, property, time, system, scale, method, and example units.
  5. Answer one scope question:
  6. Works at all laboratories: approve_universal and universal_template.
  7. Only this laboratory or test method: approve_source_specific and source_evidence.
  8. Enter the clinical rationale and submit with the authenticated reviewer identity.

fast_path_enabled, force-map controls, and raw registry JSON must not appear in this UI. A clinician approval never disables UCUM, unit/property, specimen, method, time, scale, or active-code checks.

Public Package Contract

The main project creates and persists a ReviewCase after an uncertain result:

from loinc_mapper import (
    ReviewDecision,
    PipelineSettings,
    PublicationPolicy,
    build_review_case,
    build_review_case_from_deidentified_facts,
    compile_registry_snapshot,
    correct_and_recheck,
    finalize_registry_snapshot,
    replay_registry_snapshot,
    validate_review_decision,
)

review_case = build_review_case(
    observation,
    mapping_result,
    report_reference={
        "report_id": "fax-123",
        "row_id": "row-7",
        "page": 2,
        "source_document_reference": "private://reports/fax-123/page-2",
    },
)

decision = ReviewDecision.from_dict({
    "case_id": review_case.case_id,
    "case_revision": review_case.case_revision,
    "decision": "approve_universal",
    "selected_loinc": "2345-7",
    "mapping_scope": "universal_template",
    "reviewer_id": "clinician-account-id",
    "reviewer_name": "Dr Example",
    "rationale": "Reviewed report label, mg/dL unit, and serum/plasma result style.",
})

validation = validate_review_decision(review_case, decision, catalog, active_registry)
if not validation.eligible_for_publish:
    raise ValueError(validation.errors)

policy = PublicationPolicy.CLINICIAN_FAST_REPLAY
candidate = compile_registry_snapshot(
    active_registry,
    [validation],
    catalog,
    publication_policy=policy,
)
if candidate.requires_publication:
    # mapper_factory uses PipelineSettings.clinician_review_replay().
    replay = replay_registry_snapshot(
        candidate,
        [validation],
        mapper_factory,
        publication_policy=policy,
    )
    active_snapshot = finalize_registry_snapshot(
        candidate,
        replay,
        publication_policy=policy,
    )

For a staff correction, persist the original case and decision, then use the storage-free helper to obtain the corrected facts and fresh mapper result:

recheck = correct_and_recheck(review_case, correction_decision, mapper)
# Persist recheck.corrected_observation and recheck.mapping_result as a new audit revision.

If the main project starts from already de-identified fact rows, use build_review_case_from_deidentified_facts. It accepts only the documented mapping fields and rejects patient/free-form report fields; the report reference must remain an opaque application identifier.

For normal clinician publication, mapper_factory must use PipelineSettings.clinician_review_replay(). It uses the pinned LOINC catalog and candidate registry but does not load or invoke UMLS, ScispaCy, FAISS, SapBERT, or the learned ranker. It still runs exact-alias, active-code, context, UCUM, and six-axis validation. finalize_registry_snapshot is pure: it does not write files or move an active pointer.

The candidate payload has registry_status: candidate and cannot be loaded by MappingRegistry.from_json. The finalized payload has registry_status: active and a manifest containing parent version, LOINC release, reviewer decision IDs, case IDs, checksum, replay result hash, and timestamps.

clinician_fast_replay maps all changed approvals through one map_many() batch against their original reviewed facts. A passed replay immediately enables the exact universal-template fast path. It is not a force-map: inactive codes, contained/fuzzy aliases, invalid units, conflicting specimens, and failed six-axis checks still cannot publish. Source-specific entries remain full-pipeline evidence, and clinicians never set fast-path behavior.

strict_replay remains available for release audits. It performs the original and unseen-laboratory full replays plus an equivalence fast replay, but it is not the normal clinician publication path.

Approved Alias Families

When an approval is compiled, the module generates a bounded verified_aliases list from the approved name. It contains only normalized, compact, and reviewed OCR-confusion surfaces; it does not invent semantic synonyms. Every surface remains attached to the original clinician decision and is subject to the same active-code, UCUM, specimen, method, time, scale, and six-axis checks.

An exact match to a replay-certified verified_aliases surface may receive the same clinician authority as the canonical alias. A contained display match or arbitrary fuzzy match never receives that authority and must still pass ordinary confidence and margin thresholds. This improves harmless OCR recall without turning fuzzy similarity into a clinical override.

Replay diagnostics are bounded to the top 10 candidates by default. They include status, code, confidence, margin, reasons, display names such as 718-7 - Hemoglobin [Mass/volume] in Blood, and registry evidence. They omit patient and report payloads.

Decision Values

decision Effect
approve_universal Creates a reusable source-neutral template after deterministic safety validation and one batched exact replay of the reviewed row.
approve_source_specific Creates evidence for the named laboratory/test context only. It never blocks universal retrieval for other laboratories.
correct_and_recheck Stores corrected OCR facts as an audited revision and returns them for remapping. It creates no registry entry.
reject_candidate Records why a candidate is clinically incorrect. It creates no registry entry.
not_standardized Records that no suitable structured LOINC should be emitted.
not_lab_result Records that the row does not belong in this laboratory mapper.
defer Saves no reusable decision.

Only the first two decisions accept mapping_scope. A source-specific approval requires source_laboratory. A universal approval retains the originating laboratory as provenance but does not make it a runtime prerequisite.

If a legacy source_evidence row already exists for the same alias and target, the universal approval adds a universal template and preserves the source row for audit/history. Candidate merging gives the replay-certified clinician universal entry priority over the legacy source evidence for that same code. If the existing legacy row has the same scope, publication upgrades it in the candidate snapshot instead of adding a duplicate sibling.

When a universal and source-specific entry point to different codes, both stay in the candidate union. Explicit row facts and hard validation always run first. An exact local source_test_id may select a documented local assay; otherwise, a verified clinician universal template outranks a lab-name-only source alias. A source-laboratory name by itself is retrieval evidence, never a global veto over universal mapping.

Publisher Job Status

The package emits PublicationProgressEvent values through an optional callback. The main project stores and displays these job states without holding an HTTP request open: queued, validating, batch_queued, batch_completed, finalization_ready, published, no_change, and failed_safety. The package emits the compile/replay/finalization states; the main project emits published only after it atomically advances its active registry pointer.

Only new or superseding decision revisions should be supplied to the publisher. If all submitted approvals already exist in the active snapshot, candidate.requires_publication is false and the main project records those decisions as audit-only without replaying or writing another registry object.

Specimen Contract

The UI should use select controls for canonical specimen values and preserve the original OCR wording separately:

AnalyticalSpecimen:
unknown, blood_unspecified, whole_blood, serum, plasma, serum_or_plasma,
urine, cerebrospinal_fluid, saliva, stool, sputum, amniotic_fluid,
synovial_fluid, pleural_fluid, peritoneal_fluid, body_fluid_unspecified,
tissue, other

CollectionSpecimen:
unknown, blood_venous, blood_arterial, blood_capillary,
urine_clean_catch, urine_catheter, urine_24_hour, other

Send the enum value in specimen_code or collection_specimen_code and keep the raw wording in specimen or collection_specimen. For example, collection_specimen_code=blood_venous with collection_specimen="Blood, Venous". The mapper rejects contradictory code and text instead of silently choosing one. Bld, Ser/Plas, and similar values are LOINC System-axis representations, not clinician-facing UI values.

CSV Review Workflow

The module can export an uploadable CSV with original facts as read-only context. The clinician-facing required columns are:

case_id,case_revision,decision,selected_loinc,mapping_scope,
reviewer_id,reviewer_name,rationale

Conditional columns are:

corrected_raw_name,corrected_unit,corrected_specimen,
corrected_specimen_code,corrected_collection_specimen,
corrected_collection_specimen_code,corrected_method,
corrected_time_context,corrected_scale,clear_fields,
source_test_id,source_document_reference,decision_id,reviewed_at

The exported file also includes original name, value, unit, specimen, source laboratory, report/page reference, mapper status, and candidate codes. Those are review context, not fields that a CSV import is allowed to rewrite. The full CSV also includes original_specimen_code and original_collection_specimen_code.

python -m loinc_mapper export-review-csv `
  --cases review_cases.jsonl `
  --output clinician_review_template.csv

python -m loinc_mapper validate-review-csv `
  --cases review_cases.jsonl `
  --input clinician_reviewed.csv `
  --registry config/mapping_registry.json `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --output validation_report.json

python -m loinc_mapper compile-review-snapshot `
  --cases review_cases.jsonl `
  --input clinician_reviewed.csv `
  --registry config/mapping_registry.json `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --registry-output candidate_registry.json `
  --manifest-output candidate_manifest.json

The last command produces a candidate, not a publishable runtime registry. The main project must run replay through the matching artifact bundle, call finalize_registry_snapshot, upload the finalized registry/manifest, and only then move its active pointer.

CSV uploads are revision-safe: the case revision must match the stored case, one decision is allowed per case revision in one upload, and repeated identical rows receive a deterministic upload decision ID. The main project should also enforce uniqueness of decision_id in its durable ledger.

These are recommended routes for the larger FastAPI project. They are not implemented in this module:

GET  /api/loinc/review-cases?status=open
GET  /api/loinc/review-cases/{case_id}
POST /api/loinc/review-cases/{case_id}/decisions
GET  /api/loinc/terms?query=...
POST /api/loinc/review-imports
POST /internal/loinc/registry-publications

The term-search route should call search_active_loinc(catalog, query) and return the official name, active status, six axes, example units, class, and ORDER_OBS. It must not expose inactive or order-only terms as result-row choices. The decision route should call validate_review_decision before storing an approval event. A publisher process should call compile, replay, and finalize; it should never trust a browser-submitted registry object.

Main-Project Integration Checklist

  1. Install the pinned package and artifact bundle in the mapper worker image; do not initialize SapBERT, ScispaCy, or UMLS inside the FastAPI request process.
  2. After row extraction, build LabObservation objects with every available fact and submit them in a MappingBatch. Include registry_snapshot with the finalized snapshot version, private URI, storage generation, and SHA-256.
  3. Persist mapper results and create review cases for abstain, invalid_candidate, and no_candidate. Keep report identity and document links in the main project, not in this module.
  4. Build a review screen and an optional CSV upload that only submit the versioned ReviewDecision fields described above. Staff may draft; an authorized clinician must approve reusable mappings.
  5. Use search_active_loinc for the clinician search box and validate_review_decision before writing an approval event. Return field errors to the UI instead of publishing an unsafe decision.
  6. Run a publisher worker that compiles validated events, replays candidate snapshots with the next production artifact bundle, finalizes passing snapshots, and atomically advances the active registry pointer.
  7. Launch new Cloud Run Jobs with that exact snapshot reference. A running job must finish with its own explicit reference, never reload a mutable file.

The current version does not query patient-chart pending orders. In a later version, after demographic matching, the main project may pass an advisory OrderContext containing an Elation order-set/test identifier, local lab test identifier, CPT, diagnosis, and ordering context. It must never become a hard global LOINC filter; absent, stale, or ambiguous order context falls back to normal laboratory-wide retrieval.

Recommended durable records in the main project are:

review_cases/{case_id}: immutable case, current revision, report/page reference, mapper result
review_decisions/{decision_id}: case ID/revision, reviewer identity, action, corrections, rationale
registry_publications/{version}: parent version, event IDs, manifests, replay report, active state

Keep report files, OCR artifacts, large batch payloads, results, manifests, and registry snapshot JSON in private object storage; save only their stable object references in the review ledger. The main project must enforce authorization, audit logging, retention, deletion, and incident-response policies.

Immediate Learning Versus Model Learning

There are two safe learning loops:

  1. Immediate registry learning: a finalized clinician decision appears in the next immutable registry snapshot. No neural-model training occurs.
  2. Offline model learning: retained reviewed outcomes become training and evaluation examples only after there are enough labels. A challenger ranker may become active only after a champion/challenger evaluation preserves accepted precision, unseen-laboratory performance, and zero dangerous safety errors.

The main project should create a regression case for every approved mapping, track repeat abstentions by normalized phrase and laboratory, and surface those clusters to clinicians. It must never silently turn a repeated unknown phrase into a production mapping.

Cloud Run and Audit Boundary

The deployment sequence is detailed in Deployment and release operations are in Maintenance. In short:

  1. Main API persists review cases, decisions, status history, mapper provenance, report references, and registry version references in Firestore or an equivalent durable ledger.
  2. Main API stores PDFs, OCR artifacts, batch inputs/outputs, candidate and finalized registry snapshots in private Cloud Storage.
  3. The registry publisher writes immutable object names such as mapping_registry.<version>.<sha>.json, then changes the active pointer with a Cloud Storage generation precondition.
  4. A Cloud Run Job receives a MappingBatch with an explicit registry_snapshot reference, downloads that small finalized snapshot to ephemeral storage, constructs the mapper, and verifies its loaded version.
  5. The job records snapshot URI, generation, checksum, and registry version in results. Jobs already running finish with their previous explicit snapshot.

Firestore and Cloud Storage are deployment choices, not package dependencies. Use them only under the organization's executed BAA, least-privilege IAM, audit logging, retention policy, and security review.