Data quality in medical data
Data Quality Issues in Medical Concept Normalization (MCN).
Motivation
Medical Concept Normalization connects informal health language with standardized clinical concepts. It supports pharmacovigilance, patient-record analysis, and biomedical text mining, yet much MCN research relies on single-source datasets whose quality is insufficiently examined. My work combines a broad survey of the field with a framework for improving and integrating patient-generated language data.
The survey reviews task definitions, annotation schemes, datasets, model families, benchmark performance, and applications.
Our fusion framework evaluates six social-media datasets mapped to SNOMED-CT: AskAPatient, CADEC, COMETA, PsyTAR, TwADR-S, and TwiMed. We assess correctness, concept validity, coverage, semantic variety, and class imbalance, then use Gemini with zero-shot and few-shot prompts to enrich rare concepts and fuse expressions sharing medical concepts.
The abstract reports phrase-count increases of 172.1% after augmentation and 450.0% after fusion. SapBERT, KNN-BioEL, and KRISSBERT show gains of up to 38% over single-source baselines. Concept-level macro metrics help evaluate rare cases.
Approach
Papers & outputs
MCN-1 — Diverse patient expressions across social-media datasets mapped to a shared SNOMED-CT concept, illustrating medical concept normalization and multisource fusion.
MCN-2 — Workflow for evaluating and improving dataset quality, augmenting expressions with large language models, and fusing phrases mapped to the same concept.
MCN-3 — Evaluation focuses for correctness, concept validity, coverage, semantic variety, and class imbalance.