Skip to main content

CatBoost with the scikit-learn API

This deterministic synthetic example makes categorical handling visible without a network download. It uses CatBoost's scikit-learn-compatible estimator, while cat_features and early stopping remain CatBoost-specific.

1. Create mixed-type data and three partitions​

import numpy as np
import pandas as pd
from catboost import CatBoostClassifier
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
sample_count = 360
data = pd.DataFrame(
    {
        "age": rng.integers(18, 76, size=sample_count),
        "income": rng.normal(55_000, 14_000, size=sample_count).round(0),
        "region": rng.choice(["north", "south", "west"], size=sample_count),
        "channel": rng.choice(["web", "store", "partner"], size=sample_count),
    }
)
raw_score = (
    -2.2
    + 0.035 * (data["age"] - 40)
    + 0.000025 * (data["income"] - 50_000)
    + 0.8 * (data["region"] == "north")
    + 0.6 * (data["channel"] == "web")
)
probability = 1.0 / (1.0 + np.exp(-raw_score))
target = (rng.random(sample_count) < probability).astype(int)

X_development, X_test, y_development, y_test = train_test_split(
    data,
    target,
    test_size=0.20,
    random_state=42,
    stratify=target,
)
X_train, X_valid, y_train, y_valid = train_test_split(
    X_development,
    y_development,
    test_size=0.20,
    random_state=42,
    stratify=y_development,
)

model = CatBoostClassifier(
    loss_function="Logloss",
    eval_metric="Logloss",
    iterations=400,
    learning_rate=0.05,
    depth=5,
    l2_leaf_reg=3.0,
    random_seed=42,
    thread_count=1,
    allow_writing_files=False,
    verbose=False,
)
model.fit(
    X_train,
    y_train,
    cat_features=["region", "channel"],
    eval_set=(X_valid, y_valid),
    early_stopping_rounds=25,
    use_best_model=True,
    verbose=False,
)

test_probability = model.predict_proba(X_test)[:, 1]
print("Best iteration:", model.best_iteration_)
print("Test ROC AUC:", round(roc_auc_score(y_test, test_probability), 3))
print("Test log loss:", round(log_loss(y_test, test_probability), 3))
print("Test Brier score:", round(brier_score_loss(y_test, test_probability), 3))

assert X_train.shape[0] + X_valid.shape[0] + X_test.shape[0] == sample_count
assert test_probability.shape == y_test.shape
assert 0 <= model.best_iteration_ < model.tree_count_

2. What CatBoost learns​

The strings in region and channel are declared categorical. CatBoost can form ordered category statistics without a global target encoder. This reduces a specific leakage mechanism; it does not excuse incorrect time/group splits or target-derived fields.

eval_set selects the iteration, so it is part of training. use_best_model=True retains the best prefix of trees. The isolated test set is used only for the three final metrics.

3. Tuning and evaluation​

If depth, regularization, or category settings are tuned, conduct that search inside training-only folds. CatBoost's category processing must be refitted in each fold. ROC AUC measures ranking; log loss and Brier score assess probabilities. Choose a decision threshold or calibration method on validation data, not the test set.

4. Deployment​

Store feature names/order, cat_features, category and missing-value conventions, class order, best_iteration_, model artifact, and CatBoost version. Test unknown categories and malformed schemas before serving.