Which Sentence Answers the Question?

Walkthrough

1. Load it from the catalogue

   Find the table, read its recorded target and print the column names.

   import pandas as pd
   from welt import Client
   
   with Client() as client:
       item = client.research_datasets(q="eye movements relevance", limit=1)["items"][0]
       info = client.research_dataset(item["id"])
       data = client.load_research_dataset(item["id"])
   
   target = info["targets"][0]
   target = target["name"] if isinstance(target, dict) else target
   print([column["name"] for column in info["schema"]], target)

2. Pick the rows

   Set aside 20 rows to predict at the end. The models learn from a seeded sample of 4,000 of the rest, the most this table's width allows.

   known = data.dropna(subset=[target])
   new_rows = known.sample(20, random_state=1)
   table = known.drop(new_rows.index).sample(4000, random_state=7)

3. Compare the models and predict

   The models compete on the same folds against a simple guess, then the top-ranked one is fitted on the whole table. ROC AUC runs from 0.5, a coin toss, to 1, a perfect ranking, and a lower Brier score means the probabilities can be trusted more. The last table puts each prediction next to the real answer.

   with Client() as client:
       evaluation = client.evaluate(table, target, task="classification", seed=7,
                                    models=["tabicl-v2", "kumo-medium"])
       model = evaluation.fit()
   
   print(evaluation.summary()[["name", "rank", "roc_auc_mean", "brier_mean"]])
   features = new_rows.drop(columns=target)
   print(pd.DataFrame({"actual": new_rows[target].values, "predicted": model.predict(features),
                       "probability": model.predict_proba(features).max(axis=1)}))

What you get