Which Wave Made This Noisy Signal?

Each signal is a blend of two of three simple wave shapes with random noise added. From its 21 values, which of the three classes produced it? Because the noise is known, you can see how close a model gets to the best possible score.

Use caseEngineeringPredictionModel evaluation

The table

5,000 noisy signals, each sampled at 21 points, labelled with which of three wave classes generated it. It's in Welt's research catalogue as Waveform 21, 5,000 rows by 22 columns, licensed CC BY 4.0. The catalogue measures its signal at 0.77. At 5,000 rows and 21 features the table is larger than the model limits allow, so the walkthrough evaluates and fits on a seeded sample of 4,000 rows. TabICLv2 and Kumo Tabular Medium compete.

Walkthrough

Load it from the catalogue

Find the table, read its recorded target and print the column names.

import pandas as pd
from welt import Client

with Client() as client:
    item = client.research_datasets(q="waveform 21", limit=1)["items"][0]
    info = client.research_dataset(item["id"])
    data = client.load_research_dataset(item["id"])

target = info["targets"][0]
target = target["name"] if isinstance(target, dict) else target
print([column["name"] for column in info["schema"]], target)

Pick the rows
Set aside 20 rows to predict at the end. The models learn from a seeded sample of 4,000 of the rest, the most this table's width allows.
```
known = data.dropna(subset=[target])
new_rows = known.sample(20, random_state=1)
table = known.drop(new_rows.index).sample(4000, random_state=7)
```

Compare the models and predict

The models compete on the same folds against a simple guess, then the top-ranked one is fitted on the whole table. ROC AUC runs from 0.5, a coin toss, to 1, a perfect ranking, and a lower Brier score means the probabilities can be trusted more. The last table puts each prediction next to the real answer.

with Client() as client:
    evaluation = client.evaluate(table, target, task="classification", seed=7,
                                 models=["tabicl-v2", "kumo-medium"])
    model = evaluation.fit()

print(evaluation.summary()[["name", "rank", "roc_auc_mean", "brier_mean"]])
features = new_rows.drop(columns=target)
print(pd.DataFrame({"actual": new_rows[target].values, "predicted": model.predict(features),
                    "probability": model.predict_proba(features).max(axis=1)}))

What you get

A class per signalWith a probability for each class.

A known ceilingThe best possible accuracy is known, so the gap is easy to read.

A classic benchmarkCompare with decades of published results.

Try it

Use Open with to run the research datasets notebook in Colab, Kaggle or Jupyter and search for "waveform 21", or run the code above after pip install "welt-client[parquet]".

Open with

Read the docs

Type: Use case
Area: Engineering
Question: Which Wave Made This Noisy Signal?
Model: TabICLv2, Kumo Tabular Medium
Data: Waveform 21, 5,000 rows
For: Signal-processing students and ML researchers
Licence: CC BY 4.0