Extracting adverse event nature, severity, timelines and resulting interventions from clinical notes of patients receiving CAR-T cell therapy using large language models

Chimeric Antigen Receptor T-cell (CAR‑T) therapy, genetically engineered patient T cells targeting tumor antigens, has transformed care for hematologic malignancies but requires careful tracking of adverse events (AEs) often documented only in unstructured electronic health record (EHR) notes. We evaluated a Large Language Model (LLM)-based approach in UCSF's secure environment to extract AEs, dates, grades, and interventions within 30 days post‑infusion for six commercial CAR‑T products (2012-2023), benchmarking against two evaluators. Using GPT‑4‑0314 in a zero‑shot setting with four prompts (prespecified AEs, non‑prespecified AEs, CRS, ICANS), we compared outputs against dual annotations on a random sample of 50 notes using accuracy, precision, recall, F1, and Cohen's kappa. From 4,762 progress notes for 293 patients (median age 65.6), CRS occurred in 80.2% (median onset 4 days); neutropenia 70.0% (16 days); neutropenic fever 64.8% (4 days); ICANS in 34.8%. Interventions included tocilizumab and corticosteroids. Grades were frequently undocumented (CRS 62.3%, ICANS 56.1%); documented cases were mainly CRS grade 1 (59.4%) and ICANS grade 2 (28.0%). Performance was high on CRS and ICANS grading (accuracy of 0.97 and 0.91, respectively). Moderate performances were assessed for prespecified AE extraction (accuracies 0.62-0.76), and non‑prespecified AEs (accuracies 0.76-0.84). Inter‑rater reliability (IRR) was strong for CRS/ICANS presence and grade (kappa 0.86-0.96), moderate for dates and interventions, and weaker for broader AE attributes. LLM‑derived insights can augment AE monitoring and real‑world evidence generation by unlocking unstructured clinical detail and characteristic timelines after CAR T. However, performance varied for broader AE attributes, warranting cautious use. Performance was highest for detecting and grading CRS and ICANS, with strong to near-perfect IRR. While cautious use of LLMs is warranted due to the variable performance observed in this study, these results support further evaluation of supervised CRS/ICANS extraction in controlled EHR-based research or pilot settings, with monitoring and prospective validation.