Which Heuristic Will Prove This Theorem Fastest?

A theorem prover has several search heuristics, and the right one depends on the problem. Picking it from the problem's features before the search starts can save hours of compute.

Use caseTechnologyPredictionModel evaluation

The table

2,000 first-order problems, each described by 51 numeric features, with the heuristic that solved it fastest as the label. It's in Welt's research catalogue as First-Order Theorem Proving, 2,000 rows by 52 columns, licensed CC BY 4.0. The catalogue measures its signal at 0.31. At 2,000 rows and 51 features, TabICLv2 and Kumo Tabular Medium take the table as it is. The others can't take a table this size, so the walkthrough leaves them out.

Walkthrough

Load it from the catalogue

Find the table, read its recorded target and print the column names.

import pandas as pd
from welt import Client

with Client() as client:
    item = client.research_datasets(q="first-order theorem proving", limit=1)["items"][0]
    info = client.research_dataset(item["id"])
    data = client.load_research_dataset(item["id"])

target = info["targets"][0]
target = target["name"] if isinstance(target, dict) else target
print([column["name"] for column in info["schema"]], target)

Pick the rows

Set aside 20 rows to predict at the end, and keep the rest for the models to learn from.

known = data.dropna(subset=[target])
new_rows = known.sample(20, random_state=1)
table = known.drop(new_rows.index)

Compare the models and predict

The models compete on the same folds against a simple guess, then the top-ranked one is fitted on the whole table. ROC AUC runs from 0.5, a coin toss, to 1, a perfect ranking, and a lower Brier score means the probabilities can be trusted more. The last table puts each prediction next to the real answer.

with Client() as client:
    evaluation = client.evaluate(table, target, task="classification", seed=7,
                                 models=["tabicl-v2", "kumo-medium"])
    model = evaluation.fit()

print(evaluation.summary()[["name", "rank", "roc_auc_mean", "brier_mean"]])
features = new_rows.drop(columns=target)
print(pd.DataFrame({"actual": new_rows[target].values, "predicted": model.predict(features),
                    "probability": model.predict_proba(features).max(axis=1)}))

What you get

A heuristic per problemChoose the search strategy before the prover starts.

An honest signalThe catalogue measures a modest signal here, so the baseline comparison matters.

Compute savedBetter first picks mean fewer wasted runs.

Try it

Use Open with to run the research datasets notebook in Colab, Kaggle or Jupyter and search for "first-order theorem proving", or run the code above after pip install "welt-client[parquet]".

Open with

Read the docs

Type: Use case
Area: Technology
Question: Which Heuristic Will Prove This Theorem Fastest?
Model: TabICLv2, Kumo Tabular Medium
Data: First-Order Theorem Proving, 2,000 rows
For: Researchers in automated reasoning and algorithm selection
Licence: CC BY 4.0