Which Phase of a Gesture Is the Hand In?

From the velocity and acceleration of the hands and wrists, is the speaker resting, preparing a gesture, making the stroke, holding it or returning? Splitting gestures into phases is how researchers study how people talk with their hands.

Use caseEngineeringPredictionModel evaluation

The table

2,000 frames from videos of people telling stories, with hand and wrist velocity and acceleration, labelled by gesture phase. It's in Welt's research catalogue as Gesture Phase Segmentation, 2,000 rows by 33 columns, licensed CC BY 4.0. The catalogue measures its signal at 0.34. At 2,000 rows and 32 features, TabICLv2, Kumo Tabular Medium and Mitra-v2 take the table as it is. The others can't take a table this size, so the walkthrough leaves them out.

Walkthrough

Load it from the catalogue

Find the table, read its recorded target and print the column names.

import pandas as pd
from welt import Client

with Client() as client:
    item = client.research_datasets(q="gesture phase segmentation", limit=1)["items"][0]
    info = client.research_dataset(item["id"])
    data = client.load_research_dataset(item["id"])

target = info["targets"][0]
target = target["name"] if isinstance(target, dict) else target
print([column["name"] for column in info["schema"]], target)

Pick the rows

Set aside 20 rows to predict at the end, and keep the rest for the models to learn from.

known = data.dropna(subset=[target])
new_rows = known.sample(20, random_state=1)
table = known.drop(new_rows.index)

Compare the models and predict

The models compete on the same folds against a simple guess, then the top-ranked one is fitted on the whole table. ROC AUC runs from 0.5, a coin toss, to 1, a perfect ranking, and a lower Brier score means the probabilities can be trusted more. The last table puts each prediction next to the real answer.

with Client() as client:
    evaluation = client.evaluate(table, target, task="classification", seed=7,
                                 models=["tabicl-v2", "kumo-medium", "mitra-v2"])
    model = evaluation.fit()

print(evaluation.summary()[["name", "rank", "roc_auc_mean", "brier_mean"]])
features = new_rows.drop(columns=target)
print(pd.DataFrame({"actual": new_rows[target].values, "predicted": model.predict(features),
                    "probability": model.predict_proba(features).max(axis=1)}))

What you get

A phase per frameWith a probability for each phase.

Faster annotationLabel a sample by hand and predict the rest.

An honest signalThe catalogue measures a modest signal.

Try it

Use Open with to run the research datasets notebook in Colab, Kaggle or Jupyter and search for "gesture phase segmentation", or run the code above after pip install "welt-client[parquet]".

Open with

Read the docs

Type: Use case
Area: Engineering
Question: Which Phase of a Gesture Is the Hand In?
Model: TabICLv2, Kumo Tabular Medium, Mitra-v2
Data: Gesture Phase Segmentation, 2,000 rows
For: Human-computer interaction and sign-language researchers
Licence: CC BY 4.0