OC 71.1 - Natural Language Processing for the Detection of Cancer-Associated Venous Thromboembolism

Background: Natural language processing (NLP) has been used successfully in the past to detect venous thromboembolism (VTE) in electronic medical record (EMR) corpora, but optimal performance was noted mostly on radiology reports. Application to medical notes has been less successful and targeted detection of VTE associated with cancer has not been reported to the same extent.

Aims: Assess the performance of an NLP transformer model for the task of cancer-associated VTE detection in EMR notes and radiology reports.

Methods: Training, validation, development and testing sets were derived in a 7:1:1:1 ratio from a cohort of 35,391 patients drawn from the MSK-IMPACT cohort. Clinical notes and radiology reports were labelled as occurring before or after a cancer-associated VTE event. VTE was defined as the presence of lower extremity deep vein thrombosis or pulmonary embolism. Neural network training was done using PyTorch 1.13.1 with the “longformer-large-4096” model in the Amazon Elastic Compute Cloud. Two tasks were evaluated, consisting of detecting all VTE events or detecting only cancer-associated VTE episodes. The development set was used for final assessment of model metrics.

Results: 394,948 documents retrieved from the EMR of 24,774 patients were included in the training set. Discriminatory power for detection of cancer-associated VTE at the patient level was excellent for cancer-associated VTE in the validation set (AUROC=0.974, 95% CI=0.964-0.984; see Figure). Using a detection threshold based on the validation set, sensitivity and specificity were 0.935 (0.916-0.954) and 0.974 (0.967-0.981) for the model tasked with detecting any VTE event, compared to 0.908 (0.873-0.943) and 0.976 (0.971-0.982) respectively for the model trained to detect only cancer-associated VTE (Table).

Conclusion(s): A transformer-based NLP model was derived and applied successfully to accurately identify patients with cancer-associated VTE. The loss of performance compared to detection of VTE regardless of association with cancer was not significant.