Speaker: Molu Shi, Senior Cognitive/Machine Learning Professional, Humana

Title: "Opening black-box models for better clinical intervention"

Humana’s current predictive models used to assess individuals’ risk for future non-emergent Emergency Department (ED) visit are developed using complex machine learning algorithms. Albeit their high accuracy, these models present a challenge for clinicians to decide the best intervention for a particular patient, because interpreting patient-specific clinical root causes through such complex algorithms are non-trivial.

A general algorithm, IFPI (Individual feature permutation importance) was developed to evaluate patient-specific explanations (top drivers) to predictive model risk assessment. For each driver (model feature), its impact is derived from the model prediction sensitivity to randomly permuting the driver value with another patient. For each patient, insight on root causes of model prediction can thus be derived from drivers with the highest impact ranking, i.e. top drivers. To evaluate performance of IFPI, driver impacts are averaged among all patients, the ranking of which is then compared against the population-level feature importance of the predictive model.

The IFPI algorithm was tested on an existing predictive model for non-emergent ED utilization. The model was developed using Random Forest on a historical claim data from approximately 3 million Medicare Advantage members. Driver impact of approximately 300 features used in the model were generated. In validation, Spearman correlation of 0.88 was obtained between the averaged driver impact and the model global feature importance, (Gini reduction). In production, personalized top drivers (i.e. high Behavior Health claim count, Rx non-adherence, or recent ED utilization) are listed to provide insights for outreach to high risk patients identified by the model.

Stay connected TwitterFacebookLinkedInYouTubeInstagram