Is This Patient at Risk of Heart Disease?

A clinic has a few hundred patient records and wants a risk score for each new patient. Small tables are where most models struggle, and a clinician needs to know the score is better than a guess before using it.

Use caseHealthcarePredictionModel evaluation

The table

The UCI Cleveland heart disease table has 303 patients and 13 measurements each, such as age, cholesterol, resting blood pressure and maximum heart rate. The last column records whether heart disease was found. It's small enough for every prediction model in the collection.

Walkthrough

Load the records

Read the UCI file, name the columns, and turn the outcome into yes or no.

import pandas as pd
from welt import Client

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data"
cols = ["age", "sex", "cp", "trestbps", "chol", "fbs", "restecg",
        "thalach", "exang", "oldpeak", "slope", "ca", "thal", "disease"]
patients = pd.read_csv(url, names=cols, na_values="?")
patients["disease"] = (patients["disease"] > 0).map({True: "yes", False: "no"})
print(patients.shape, patients["disease"].value_counts().to_dict())

Pick the patients
Set aside 20 patients to score at the end, and keep the rest for the models to learn from.
```
new_rows = patients.sample(20, random_state=1)
table = patients.drop(new_rows.index)
```

Test all four models and score

On a table this small, every prediction model in the collection can take part, and the baseline line shows what a guess would score. ROC AUC runs from 0.5, a coin toss, to 1, a perfect ranking, and a lower Brier score means the probabilities can be trusted more. The last table puts each patient's risk next to their diagnosis.

with Client() as client:
    evaluation = client.evaluate(table, "disease", task="classification", seed=7,
                                 models=["tabicl-v2", "kumo-medium", "mitra-v2", "tabdpt-1.3"])
    model = evaluation.fit()

print(evaluation.summary()[["name", "rank", "roc_auc_mean", "brier_mean"]])
features = new_rows.drop(columns="disease")
print(pd.DataFrame({"actual": new_rows["disease"].values,
                    "risk": model.predict_proba(features)[:, 1]}))

What you get

A risk score per patientA probability, not just a yes or no.

Proof it beats a guessEvery model is ranked against the baseline on records it never saw.

The right model for small dataThe leaderboard shows which model handles 303 rows best.

Try it

Use Open with to run the evaluation notebook in Colab, Kaggle or Jupyter, or run the code above after pip install "welt-client[parquet]". For a clinical study, report the model using the TRIPOD+AI checklist.

Open with

Read the docs

Type: Use case
Area: Healthcare
Question: Is This Patient at Risk of Heart Disease?
Model: TabICLv2, Kumo Tabular Medium, Mitra-v2, TabDPT 1.3
Data: UCI Heart Disease (Cleveland), 303 patients
For: Clinical researchers and health data scientists
Licence: CC BY 4.0