Skip to content

Performance, Profiles, and Worker Batches

The mapper is intentionally safety-first. Performance work removes duplicated computation; it never removes the clinical checks that decide whether a LOINC code can be emitted.

Production Flow

For every observation that is not a verified exact universal-template fast-path case:

normalize
-> AxisFactExtractor + UniversalSemanticIndex
-> deterministic SQLite retrieval
-> ScispaCy mention and abbreviation processing
-> local UMLS CUI retrieval
-> FAISS/SapBERT semantic retrieval
-> UCUM and six-axis validation
-> feature ranking
-> confidence and margin gate

The only permitted shortcut is an exact reviewed universal_template alias with review_status=verified_active and fast_path_enabled=true. It still proves:

  • the target code is active in the pinned LOINC release;
  • declared unit, class, property, method, and specimen requirements are met;
  • UCUM/unit and six-axis validation pass;
  • no inferred context conflicts exist; and
  • the confidence policy accepts the single reviewed target.

Fast-path provenance lists skipped semantic stages. A contained alias, legacy registry row, source-specific evidence record, unknown unit, or missing required specimen/method always runs the full pipeline.

Settings

Copy .env.example to the ignored .env file and keep production settings enabled:

PIPELINE_PROFILE=production
IS_UMLS_ON=true
IS_FAISS_ON=true
IS_SAPBERT_ON=true
IS_VALIDATION_ON=true
IS_CONFIDENCE_ON=true
IS_REGISTRY_FAST_PATH_ON=true
MAPPER_CPU_THREADS=4

production rejects any disabled evidence stage. evaluation is the only profile that allows controlled ablations. Any evaluation result is marked execution.experimental=true, and ElationAdapter refuses to create a payload from it.

Validation and confidence cannot be disabled in any profile. Do not use a profile experiment as a deployment setting.

Why It Is Faster

  • The SQLite catalog loads once per worker. Class/type membership and token statistics are kept in memory for O(1) scope checks.
  • The release-derived universal component/axis index is compiled once from that loaded catalog. It creates source-neutral candidates from facts such as urine, CSF, 24-hour duration, and stated methods; it does not query a per-laboratory mapping table for ordinary rows.
  • Deterministic lookup uses the indexed alias_tokens table first. FTS5 is a bounded prefix fallback, not a broad query over every matching alias.
  • Character recovery uses distributed 3-of-4 trigram intersections. It still tolerates a single damaged OCR trigram, but avoids the old broad trigram OR scan for words such as cholesterol.
  • Deterministic work and context extraction are cached per normalized report row before individual safety decisions.
  • ScispaCy processes a report through nlp.pipe; parser/tagger components not needed for linking are disabled.
  • The UMLS linker owns one read-only SQLite connection and caches mention-to-CUI results for the worker lifetime.
  • A linked UMLS name receives bounded exact/token local expansion rather than replaying the complete OCR deterministic pipeline once for every sibling CUI. Direct CUI-to-LOINC bridge candidates are always retained.
  • SapBERT encodes unique queries in batches. FAISS searches are batched too.
  • Code-level SapBERT vectors avoid re-embedding candidate descriptions during reranking.
  • Optional class shards search an official LOINC class first, then use a laboratory/global fallback if a shard produces no scoped hit.

Every result records provenance.stages.timings_ms, candidate counts, profile, artifact versions, cache hits, and fallback evidence. Inspect those fields before changing a threshold.

The focused real-catalog benchmark after this change was 611.6 ms for deterministic retrieval of VLDL Cholesterol Cal, versus the earlier 40-48-second hot path. This is not an end-to-end clinical latency claim; ScispaCy, UMLS, FAISS, validation, and ranking must still be measured on the frozen production corpus after the serving artifact is built.

The raw 2026AA fallback now performs intersection-based token lookup and measured 5.3 s for that same UMLS name, down from 17.3 s; it is still too slow for the report target. The compact serving artifact remains a required deployment gate for production latency measurements.

UMLS Serving Artifact

The raw UMLS umls.sqlite3 may be large because it contains broad Metathesaurus and older trigram structures. Build this compact runtime artifact once after verifying raw assets:

python -m loinc_mapper build-umls-serving `
  --asset-path assets/umls/2026AA `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --output assets/umls/2026AA/serving/umls_loinc.sqlite3

It retains UMLS aliases only when their CUI bridges to an active local LOINC code and creates exact, FTS5, and trigram indexes. The source release is validated by checksum during this build. At worker startup, the serving manifest validates release, artifact shape, and file size without rehashing the raw 14 GB file. Runtime provenance should then show serving_exact+fts5+trigram.

Keep raw licensed data private and immutable so the serving artifact can be rebuilt and audited later.

For the background PowerShell build, follow progress on standard error:

Get-Content evaluation\logs\umls_serving_build.err.log -Wait

FAISS Derived Artifacts

An existing terms.faiss can gain code-level vectors without a new SapBERT encoding run:

python -m loinc_mapper build-code-vectors `
  --index assets/loinc/2.82/terms.faiss `
  --metadata assets/loinc/2.82/terms.json

Optional laboratory/class shards are also derived from the same global index:

python -m loinc_mapper build-vector-shards `
  --catalog assets/loinc/2.82/catalog_v5.sqlite3 `
  --index assets/loinc/2.82/terms.faiss `
  --metadata assets/loinc/2.82/terms.json `
  --output assets/loinc/2.82/shards

Shards are a latency optimization only. SQLite remains the authoritative LOINC catalog, and every shard result still undergoes the same laboratory-type, UCUM, and six-axis safety checks. Class agreement is advisory evidence; a laboratory-wide fallback prevents a mixed report section from erasing a valid candidate. Benchmark class shards before shipping them because they consume additional disk space.

Batch Worker Contract

The main fax project should submit up to 20 reports or 500 observations to one long-lived worker process. MappingBatch version 1 looks like this:

{
  "schema_version": "1",
  "batch_id": "fax-batch-2026-07-31-001",
  "output_location": "gs://private-bucket/results/batch-001.json",
  "registry_snapshot": {
    "registry_version": "registry-20260807T0100-abc123",
    "uri": "gs://private-bucket/registries/mapping_registry.registry-20260807T0100-abc123.json",
    "generation": "1722980000000000",
    "sha256": "..."
  },
  "reports": [
    {
      "report_id": "fax-123",
      "observations": [
        {"raw_name": "LDL Cholesterol", "value": "133", "unit": "mg/dL", "loinc_class_hint": "CHEM"}
      ]
    }
  ]
}

Run it locally with:

python -m loinc_mapper run-worker `
  --input batch.json `
  --output batch-result.json `
  --umls-path assets/umls/2026AA

The worker creates one mapper, calls map_many() over the flattened batch, then restores report boundaries in the JSON result. The main project owns any queue acknowledgement or callback delivery; it should persist the full result before attempting Elation submission. It must download and verify the explicit active registry snapshot before constructing the mapper; BatchWorker verifies that the loaded registry version matches the batch reference.

Profile Matrix

Use one frozen, expert-reviewed corpus to compare the deployed path with diagnostic profiles:

python -m loinc_mapper evaluate `
  --input evaluation/gold/holdout.csv `
  --details evaluation/gold/profile_matrix.json `
  --profile-matrix `
  --umls-path assets/umls/2026AA

The report includes accepted precision, top-1/top-3, coverage, abstention, dangerous false positives, per-stage mean/p50/p95 latency, and fast-path equivalence against the full pipeline. Do not activate a change that loses a known-correct regression mapping, reduces accepted precision, or creates a dangerous false positive.

Cloud Run Job Baseline

Start with a scale-to-zero Cloud Run Job, one mapper process per task, 4 vCPU and 8 GiB. Batch reports for up to 60 seconds or 20 reports / 500 rows before submitting one task. Package only compact serving artifacts; mount or fetch raw licensed UMLS data from private storage only for rebuild/audit jobs.

The pinned SapBERT files must be present in the worker image or mounted model volume before startup. The mapper uses local-only model loading so a production task never silently downloads a model from Hugging Face. On Windows development hosts, configure a sufficiently large paging file; a paging-file allocation failure is an infrastructure issue, not a reason to substitute an unvalidated smaller model.

Do not introduce Redis or ClickHouse until measurements show that multiple warm workers need a shared cache. SQLite plus FTS5, FAISS, and the in-process LRU keep this module simple enough to operate within the initial cost target.