Disparate Model Performance and Stability in Machine Learning Clinical Support for Diabetes and Heart Diseases

Area of Science:

• Biomedical Informatics
• Machine Learning
• Health Equity

Background:

• Machine learning (ML) algorithms are increasingly used for clinical decision-making.
• Underrepresentation of marginalized groups in training data can lead to demographic inequities in ML model performance.
• Existing chronic disease datasets exhibit sex- and age-related disparities.

Purpose of the Study:

• To investigate sex- and age-related inequities in chronic disease datasets and derived ML models.
• To introduce a novel analytical framework combining systematic arbitrariness with traditional performance metrics.
• To evaluate the impact of data complexity on model performance across different demographic groups.

Main Methods:

• Analysis of chronic disease data from over 25,000 individuals.
• Application of a novel framework assessing systematic arbitrariness alongside accuracy and data complexity.
• Evaluation of ML model performance across sex and age demographics.

Main Results:

• Mild sex-related disparities were observed, with higher predictive accuracy for males.
• Significant age-related differences showed better accuracy for younger patients.
• Older patients exhibited inconsistent predictive accuracy across datasets, correlated with increased data complexity and reduced model performance.

Conclusions:

• Training data representativeness alone does not ensure equitable ML outcomes in healthcare.
• Systematic arbitrariness in ML models contributes to demographic performance variations.
• Addressing model arbitrariness is essential before clinical deployment to ensure fairness.