How Badly Would an Earthquake Damage This Building?

From its age, height, footprint, materials and use, which damage grade is this building likely to reach in an earthquake? Planners want to know which buildings to inspect or strengthen before the next one.

Use caseEngineeringPredictionModel evaluation

The table

Buildings surveyed after an earthquake, with region, number of floors, age, area, structure, materials and use, and the damage grade each one suffered. It's in Welt's research catalogue as BMC Training Data: Building Characteristics, 177,640 rows by 40 columns, licensed CC0. The catalogue measures its signal at 0.35. At 177,640 rows and 39 features the table is larger than the model limits allow, so the walkthrough evaluates and fits on a seeded sample of 4,000 rows. TabICLv2 and Kumo Tabular Medium compete.

Walkthrough

Load it from the catalogue

Find the table, read its recorded target and print the column names.

import pandas as pd
from welt import Client

with Client() as client:
    item = client.research_datasets(q="BMC building characteristics", limit=1)["items"][0]
    info = client.research_dataset(item["id"])
    data = client.load_research_dataset(item["id"])

target = info["targets"][0]
target = target["name"] if isinstance(target, dict) else target
print([column["name"] for column in info["schema"]], target)

Pick the rows
Set aside 20 rows to predict at the end. The models learn from a seeded sample of 4,000 of the rest, the most this table's width allows.
```
known = data.dropna(subset=[target])
new_rows = known.sample(20, random_state=1)
table = known.drop(new_rows.index).sample(4000, random_state=7)
```

Compare the models and predict

The models compete on the same folds against a simple guess, then the top-ranked one is fitted on the whole table. ROC AUC runs from 0.5, a coin toss, to 1, a perfect ranking, and a lower Brier score means the probabilities can be trusted more. The last table puts each prediction next to the real answer.

with Client() as client:
    evaluation = client.evaluate(table, target, task="classification", seed=7,
                                 models=["tabicl-v2", "kumo-medium"])
    model = evaluation.fit()

print(evaluation.summary()[["name", "rank", "roc_auc_mean", "brier_mean"]])
features = new_rows.drop(columns=target)
print(pd.DataFrame({"actual": new_rows[target].values, "predicted": model.predict(features),
                    "probability": model.predict_proba(features).max(axis=1)}))

What you get

A damage grade per buildingWith a probability for each grade.

Where to inspect firstRank buildings by their chance of heavy damage.

An honest signalThe catalogue measures a modest signal here, so the comparison with a simple guess matters.

Try it

Use Open with to run the research datasets notebook in Colab, Kaggle or Jupyter and search for "BMC building characteristics", or run the code above after pip install "welt-client[parquet]".

Open with

Read the docs

Type: Use case
Area: Engineering
Question: How Badly Would an Earthquake Damage This Building?
Model: TabICLv2, Kumo Tabular Medium
Data: BMC Training Data: Building Characteristics, 177,640 rows
For: Structural engineers and disaster planners
Licence: CC0