The Evidence Locked Inside the Chart
Most of what a clinician actually observes never makes it into a structured field. Tumor response, symptom trajectory, functional status, the reasoning behind a treatment switch: this detail lives in free-text progress notes, radiology reports, pathology summaries, and discharge letters. Structured fields like diagnosis codes and lab values capture only a fraction of the clinical picture, and estimates across the informatics literature routinely put the unstructured share of the electronic health record (EHR) at the majority of its clinical content.
For pharmaceutical and life sciences organizations, that trapped information is exactly what real-world evidence (RWE) programs need: outcomes, disease progression, comorbidities, and treatment patterns observed in routine care rather than in a controlled trial. Regulators have made clear they expect to see it used well. The U.S. Food and Drug Administration's 2024 guidance, "Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products," sets an explicit bar: sponsors must demonstrate data traceability and quality "during data accrual, data curation, and incorporation into the final study-specific dataset," and should apply quantitative bias analysis where misclassification could affect study conclusions. Extracting reliable variables from unstructured text at scale, without introducing exactly that misclassification, is the hard problem RWE teams now have to solve.
How NLP Pipelines Turn Notes into Structured Evidence
A production-grade clinical NLP pipeline is not a single model but a chain of steps, each with its own failure modes to manage. Named entity recognition locates clinical concepts (a tumor site, a lab result mentioned in prose, a medication). Relation and event extraction link those concepts together and place them in time. Negation and assertion detection distinguish a finding that is present from one the note explicitly rules out, a distinction that changes a variable's value entirely. Increasingly, large language models are used for abstraction tasks that once required a human reviewer, such as characterizing disease progression from a sequence of oncology notes.
What separates a research prototype from an evidence source a regulator or payer will accept is validation. Flatiron Health, a real-world oncology data company, built its VALID framework (Validation of Accuracy for LLM/ML-Extracted Information and Data) specifically to compare model output against primary and duplicate human abstractors on the same charts, and to keep monitoring performance after deployment rather than validating once and assuming stability. That ongoing discipline, comparing extracted variables against a human-curated reference standard, tracking sensitivity, specificity, and positive predictive value over time, and re-validating when the underlying documentation practices shift, is what turns an NLP pipeline into a defensible evidence-generation system rather than a black box.
- Curated, clinically informed reference standards built by trained abstractors, used to train and continuously monitor extraction models
- Structured and unstructured sources reconciled together (EHR notes, claims, registries) rather than treated as separate data silos
- Documented validation metrics and drift monitoring, so performance changes are caught before they reach a submission or a payer dossier
What the Evidence Shows So Far
The published record is encouraging but disciplined about its limits. Castellanos, Wittmershaus, and Chandwani, writing in JCO Clinical Cancer Informatics (2024), argue that oncology real-world data quality "is not a single concept" and that machine learning approaches can help unlock scale, but only if they are validated and monitored against abstracted reference data with stable sensitivity, specificity, and positive predictive value over time. In 2025, Flatiron Health reported that an Anthropic-powered large language model extracted cancer progression events across 14 tumor types with F1 performance comparable to expert human abstractors, and produced real-world progression-free survival estimates nearly identical to those derived by people.
"Scalable, high-quality extraction of such an important and complex endpoint like progression will open new doors for novel research," said Aaron B. Cohen, the study's lead author, as reported by Medical Economics.
Independent systematic reviews add useful context on variability. A 2025 review in JMIR Medical Informatics examining NLP for cancer-related information extraction from EHRs, and a 2024 review in JAMIA Open covering NLP systems for extracting activities-of-daily-living information, both describe a field where performance is genuinely improving but still varies substantially by clinical concept, note type, and health system, underscoring why validation cannot be a one-time exercise performed on a single dataset.
Building This Responsibly: Where ANG Associates Fits
Turning promising NLP research into a pipeline a regulatory affairs or market access team can actually rely on is primarily an engineering, quality, and governance problem, not a modeling one. ANG Associates works with pharma and life sciences organizations across these layers: defining an AI strategy that identifies which extraction tasks are mature enough for production use today versus those still needing human-in-the-loop review, engineering the data pipelines that reconcile EHR, claims, and registry sources while preserving traceability back to source documents, and applying GxP-aligned validation practices, including computer system validation and continued performance monitoring, to the NLP components that feed regulatory submissions or HEOR analyses. Because these pipelines process identifiable patient-level health data, compliance with the EU GDPR and Switzerland's revised Federal Act on Data Protection (nFADP/nDSG) has to be designed in from the start, covering data residency, purpose limitation, and the special protections that apply to health data, rather than addressed as an afterthought once a model is already in production. Delivered through a SAFe or Agile framework, this work stays iterative and auditable at the same time, which is precisely what regulatory-grade real-world evidence demands.
Sources
- "Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products," Guidance for Industry, U.S. Food and Drug Administration, Federal Register notice, 2024.
- Castellanos E, Wittmershaus B, Chandwani S. "Raising the Bar for Real-World Data in Oncology: Approaches to Quality Across Multiple Dimensions." JCO Clinical Cancer Informatics, 2024.
- "Flatiron Health study shows AI can match human experts in tracking cancer progression." Medical Economics, 2025.
- "Performance of Natural Language Processing for Information Extraction From Electronic Health Records Within Cancer: Systematic Review." JMIR Medical Informatics, 2025.
- "Natural Language Processing Systems for Extracting Information From Electronic Health Records About Activities of Daily Living: A Systematic Review." JAMIA Open, 2024.
- Swiss Clinical Trial Organisation (SCTO). "The New Federal Act on Data Protection (nFADP): Guidance for Clinical Trials," 2023.