Welcome to CatBoost! 🚀 In this comprehensive guide, we'll explore CatBoost (Categorical Boosting) - a powerful gradient boosting framework developed by Yandex that excels at handling categorical features and uses symmetric trees for better generalization. Think of it as the "smart handler" of gradient boosting!
- What is CatBoost?
- How CatBoost Works
- The Mathematical Foundation
- Implementation Details
- Step-by-Step Example
- Real-World Applications
- Understanding the Code
- Model Evaluation
CatBoost (Categorical Boosting) is a gradient boosting framework developed by Yandex that handles categorical features naturally and uses symmetric (oblivious) trees. It addresses critical issues like prediction shift through ordered boosting, making it highly robust and accurate.
Real-world analogy: If XGBoost is a meticulous craftsman and LightGBM is a speed demon, CatBoost is like a wise architect who:
- Builds symmetric, balanced structures (oblivious trees)
- Prevents contamination (ordered boosting avoids target leakage)
- Handles different materials naturally (categorical features)
- Focuses on stability and reliability
| Aspect | Details |
|---|---|
| Algorithm Type | Ensemble Learning (Advanced Boosting) |
| Learning Style | Supervised Learning |
| Primary Use | Regression, Classification, Ranking |
| Base Learners | Symmetric (oblivious) decision trees |
| Key Innovation | Ordered boosting + Symmetric trees + Categorical handling |
"CatBoost = Gradient Boosting + Symmetric Trees + Ordered Boosting + Categorical Intelligence"
CatBoost improves upon XGBoost and LightGBM through:
- Symmetric trees: All nodes at same level use same split
- Ordered boosting: Prevents prediction shift and target leakage
- Ordered target statistics: Smart encoding for categorical features
- Robust defaults: Works well out-of-the-box
- Handles categoricals natively: No need for one-hot encoding
1. Tree Structure: Symmetric vs Asymmetric
XGBoost (Level-wise): LightGBM (Leaf-wise): CatBoost (Symmetric):
Root Root Root
/ \ / \ / \
A B A B A B
/ \ / \ / \ [Feature 2] [Feature 2]
C D E F C D / \ / \
/ \ C D E F
E F
Balanced tree Asymmetric tree Symmetric tree
All level splits Best leaf split Same split at level
different different SAME for both!
Key difference: In CatBoost, both A and B split on the SAME feature with SAME threshold!
2. Symmetric (Oblivious) Trees
Traditional Trees:
- Each node can split on any feature
- Flexible but complex
- Hard to regularize
CatBoost Symmetric Trees:
- All nodes at level L split on same feature & threshold
- Simpler structure
- Natural regularization
- Faster prediction: O(depth) instead of O(depth × branching)
Example with depth=3:
Level 0: ALL split on "Age <= 30"
Level 1: ALL split on "Income <= 50K"
Level 2: ALL split on "Score <= 700"
Result: 2^3 = 8 leaves
Prediction path: Just check 3 conditions → get leaf index!
3. Ordered Boosting
Problem with traditional boosting:
- When fitting tree T, use predictions from trees 1..T-1
- But trees 1..T-1 were trained on the SAME data
- This causes PREDICTION SHIFT and TARGET LEAKAGE
- Model sees labels during gradient calculation
CatBoost's Solution: Ordered Boosting
- Divide data into random permutations
- For each sample, use predictions from models trained on OTHER samples
- Prevents target leakage
- More robust, less overfitting
Simplified example:
Training samples: [1, 2, 3, 4, 5]
- For sample 3's gradient: use model trained only on {1, 2}
- For sample 5's gradient: use model trained only on {1, 2, 3, 4}
- Never use same sample for both training and gradient calculation!
4. Categorical Feature Handling
XGBoost/LightGBM:
- Need to encode categoricals manually
- One-hot encoding (explodes features)
- Label encoding (loses information)
- Target encoding (risk of target leakage)
CatBoost:
- Handles categoricals NATIVELY
- Uses "Ordered Target Statistics"
- Computes target mean for each category
- But in special order to prevent leakage
- Automatically optimal encoding
Example: Color = ["Red", "Blue", "Green"]
CatBoost internally: Red→0.65, Blue→0.42, Green→0.78
(based on target statistics, not arbitrary numbers!)
5. Default Learning Rate
XGBoost: 0.3 (aggressive)
LightGBM: 0.1 (moderate)
CatBoost: 0.03 (conservative)
Why CatBoost uses lower rate?
- Symmetric trees are simpler
- Ordered boosting adds complexity
- Lower rate + more trees = better generalization
Step 1: Quantize numerical features into discrete bins
- Similar to LightGBM's histogram
- Typical: 128 borders per feature
↓
Step 2: Handle categorical features (if any)
- Convert to ordered target statistics
- Prevents target leakage through ordering
↓
Step 3: Initialize predictions (base_score)
↓
Step 4: For each boosting iteration:
a. Calculate gradients
b. Apply ordered boosting (simplified in basic implementation)
↓
Step 5: Build SYMMETRIC tree:
- For each level (depth):
* Try all features and thresholds
* Pick split that ALL nodes at this level will use
* Split ALL current partitions with this split
↓
Step 6: Calculate leaf values with L2 regularization:
value = -sum(gradients) / (count + l2_leaf_reg)
↓
Step 7: Update predictions: F(x) = F(x) + η × tree(x)
↓
Repeat Steps 4-7 for n_estimators
Let's predict loan default using symmetric trees:
Data:
Customer | Income | Debt | Existing_Loans | Default?
---------|--------|------|----------------|----------
A | 50 | 20 | 1 | 0
B | 80 | 15 | 0 | 0
C | 40 | 35 | 2 | 1
D | 90 | 10 | 0 | 0
E | 35 | 40 | 3 | 1
F | 70 | 25 | 1 | 0
G | 45 | 38 | 2 | 1
H | 95 | 8 | 0 | 0
Step 1: Quantize Features
Income bins (border_count=2):
Bin 0: Income ≤ 60 → [50, 40, 35, 45]
Bin 1: Income > 60 → [80, 90, 70, 95]
Debt bins (border_count=2):
Bin 0: Debt ≤ 22.5 → [20, 15, 10, 8]
Bin 1: Debt > 22.5 → [35, 40, 25, 38]
Existing_Loans bins (border_count=2):
Bin 0: Loans ≤ 1.5 → [1, 0, 0, 1, 0]
Bin 1: Loans > 1.5 → [2, 3, 2]
Step 2: Initialize
Default rate: p = 3/8 = 0.375
base_score = log(0.375 / 0.625) = log(0.6) = -0.51
Initial predictions (log-odds): [-0.51] × 8
Initial probabilities: sigmoid(-0.51) = 0.375 for all
Step 3: Calculate Gradients
For binary log loss:
g = p - y
Customer | y | p=0.375 | gradient
---------|---|---------|----------
A | 0 | 0.375 | 0.375
B | 0 | 0.375 | 0.375
C | 1 | 0.375 | -0.625
D | 0 | 0.375 | 0.375
E | 1 | 0.375 | -0.625
F | 0 | 0.375 | 0.375
G | 1 | 0.375 | -0.625
H | 0 | 0.375 | 0.375
Gradient array: [0.375, 0.375, -0.625, 0.375, -0.625, 0.375, -0.625, 0.375]
Step 4: Build Symmetric Tree (Depth=2)
LEVEL 0: Choose split for ALL root partitions
Try Income <= Bin 0 (Income ≤ 60):
Left: [A, C, E, G] gradients = [0.375, -0.625, -0.625, -0.625]
Right: [B, D, F, H] gradients = [0.375, 0.375, 0.375, 0.375]
Calculate gain (with l2_leaf_reg=3):
Left: G_L = -1.5, count = 4
Score_L = (-1.5)² / (4 + 3) = 2.25 / 7 = 0.321
Right: G_R = 1.5, count = 4
Score_R = (1.5)² / (4 + 3) = 2.25 / 7 = 0.321
Parent: G_P = 0, count = 8
Score_P = 0² / (8 + 3) = 0
Gain = 0.321 + 0.321 - 0 = 0.642 ← Best split!
Decision: Level 0 splits on "Income <= 60"
Current state:
├─ Partition 0 (Low income): [A, C, E, G]
└─ Partition 1 (High income): [B, D, F, H]
LEVEL 1: Choose ONE split for BOTH partitions
Try Debt <= Bin 0 (Debt ≤ 22.5):
For Partition 0 (Low income):
Left (Low income, Low debt): [A] gradients = [0.375]
Right (Low income, High debt): [C, E, G] gradients = [-0.625, -0.625, -0.625]
For Partition 1 (High income):
Left (High income, Low debt): [B, D, H] gradients = [0.375, 0.375, 0.375]
Right (High income, High debt): [F] gradients = [0.375]
Calculate total gain across BOTH partitions:
Gain_partition0 + Gain_partition1 = 0.45 + 0.12 = 0.57
This is the best split across all features!
Decision: Level 1 splits on "Debt <= 22.5"
Final tree structure (Symmetric!):
[Income <= 60]
/ \
[Debt <= 22.5] [Debt <= 22.5]
/ \ / \
Leaf0 Leaf1 Leaf2 Leaf3
[A] [C,E,G] [B,D,H] [F]
Step 5: Calculate Leaf Values
Leaf 0 (Low income, Low debt): [A]
G = 0.375, count = 1
value = -0.375 / (1 + 3) = -0.094
Leaf 1 (Low income, High debt): [C, E, G]
G = -1.875, count = 3
value = -(-1.875) / (3 + 3) = 0.313
Leaf 2 (High income, Low debt): [B, D, H]
G = 1.125, count = 3
value = -1.125 / (3 + 3) = -0.188
Leaf 3 (High income, High debt): [F]
G = 0.375, count = 1
value = -0.375 / (1 + 3) = -0.094
Notice how L2 regularization (3.0) shrinks values toward zero!
Step 6: Update Predictions
Learning rate η = 0.05
Customer A: -0.51 + 0.05 × (-0.094) = -0.515
Customer C: -0.51 + 0.05 × 0.313 = -0.494
Customer B: -0.51 + 0.05 × (-0.188) = -0.519
...
After 100 trees:
High-risk customers (C, E, G) → positive log-odds → p > 0.5
Low-risk customers (A, B, D, F, H) → negative log-odds → p < 0.5
Why Symmetric Trees Help:
Advantages:
1. Regularization: Simpler structure prevents overfitting
2. Fast prediction: Just check depth conditions
3. Interpretability: Easy to understand decision path
4. Robustness: Less sensitive to noise
Prediction for new customer:
- Income = 55 → Goes left (≤ 60)
- Debt = 30 → Goes right (> 22.5)
- Leaf index: 01 (binary) = 1 → Leaf 1
- Prediction: Add leaf 1 value from each tree!
Traditional tree: O(depth × branches) comparisons
Symmetric tree: O(depth) comparisons → Much faster!
CatBoost optimizes a regularized objective similar to XGBoost:
Obj = Σ L(yᵢ, ŷᵢ) + Σ Ω(fₜ)
Where:
- L(yᵢ, ŷᵢ) = loss function (RMSE for regression, logloss for classification)
- Ω(fₜ) = regularization for tree t
- Ω(f) = γT + λΣ(w²ᵢ)
- γ: penalty for number of leaves (implicit through depth)
- λ: L2 regularization on leaf weights (l2_leaf_reg)
- T: number of leaves = 2^depth
CatBoost uses first-order gradients:
g = ∂L/∂ŷ
For squared loss (L2):
L = ½(y - ŷ)²
g = ŷ - y
For log loss (binary classification):
L = -[y·log(p) + (1-y)·log(1-p)]
where p = sigmoid(ŷ) = 1/(1 + e^(-ŷ))
g = p - y
Why only first-order?
- Simpler computation
- Ordered boosting provides enough regularization
- Symmetric trees add natural regularization
- Still achieves excellent performance
For a symmetric tree, all nodes at level L use the same split:
At each level, find split (feature, threshold) that maximizes:
Gain = Σ [Loss_after(partition_i) - Loss_before(partition_i)]
for all current partitions
Where for each partition:
Loss = -G² / (N + λ)
- G = sum of gradients in partition
- N = number of samples in partition
- λ = l2_leaf_reg
Process:
1. Start with all data as one partition
2. For each level:
- Try all features and thresholds
- Evaluate: if this split is applied to ALL partitions, what's total gain?
- Pick best overall split
- Apply it to ALL partitions → double the partitions
3. After depth levels: have 2^depth partitions (leaves)
Optimal leaf value with L2 regularization:
w* = -G / (N + λ)
Where:
- G = Σ gᵢ for samples in leaf
- N = number of samples in leaf
- λ = l2_leaf_reg (default: 3.0)
Interpretation:
- Without regularization (λ=0): w = -G/N (simple average)
- With regularization: w is shrunk toward zero
- Small leaves (small N): more shrinkage
- Large leaves (large N): less shrinkage
Example:
Leaf with 10 samples, G = -5.0, λ = 3.0
w = -(-5.0) / (10 + 3) = 5.0 / 13 = 0.385
Same gradient with 2 samples:
w = 5.0 / (2 + 3) = 5.0 / 5 = 1.0
→ Smaller leaf gets more shrinkage!
Fast prediction using binary indexing:
For a tree with depth D:
1. Initialize leaf_index = 0
2. For each level l from 0 to D-1:
a. Check split condition at level l
b. If condition FALSE (goes right):
leaf_index += 2^(D-l-1)
3. Return leaf_value[leaf_index]
Example: depth = 3
Level 0: Income <= 60? → NO → leaf_index += 4 = 4
Level 1: Debt <= 20? → YES → leaf_index += 0 = 4
Level 2: Loans <= 1? → NO → leaf_index += 1 = 5
→ Leaf index = 5 → return leaf_value[5]
Complexity: O(depth) per sample
Compare to traditional tree: O(depth × log(features)) average case
CatBoost addresses prediction shift:
Problem: Traditional Boosting
- Fit tree T using gradients from model M_{T-1}
- But M_{T-1} was trained on the SAME data
- Model has seen the labels during training
- Causes overfitting and prediction shift
Solution: Ordered Boosting
- Use multiple random permutations of data
- For sample i: calculate gradient using model trained only on samples BEFORE i
- Prevents target leakage
Simplified algorithm:
1. Create random permutation σ of training data
2. For sample at position i in σ:
- Use model M_i trained only on σ[0:i]
- Calculate gradient g_i using M_i
3. Build tree using these unbiased gradients
Full CatBoost implementation:
- Uses multiple permutations
- Maintains multiple models
- Complex but prevents overfitting
Our simplified version:
- Standard boosting with strong regularization
- Still effective with symmetric trees + L2 reg
Smart categorical encoding to prevent leakage:
Problem: Simple target encoding
- For category C: encode as mean(target | category = C)
- But this uses the SAME samples' targets
- Target leakage! Model has seen the answer
CatBoost's Solution: Ordered Target Statistics
1. Create random permutation of data
2. For sample i with category C:
- Encode C as mean of target for samples with C that appear BEFORE i
- Add prior (smooth with global mean)
Formula:
OTS(x_i) = (countPrior × prior + Σ y_j) / (countPrior + count)
Where:
- Sum over j: samples with same category BEFORE i in permutation
- prior: global mean target
- countPrior: smoothing parameter (typically 1-10)
Example:
Category "Red" appears at positions: 3, 7, 12, 18
Targets: 1, 0, 1, 1
Prior = 0.5, countPrior = 1
Position 3 (first Red):
OTS = (1×0.5 + 0) / (1 + 0) = 0.5 (only prior)
Position 7 (second Red):
OTS = (1×0.5 + 1) / (1 + 1) = 0.75 (prior + first Red's target)
Position 12 (third Red):
OTS = (1×0.5 + 1+0) / (1 + 2) = 0.5 (prior + first two Reds)
Position 18 (fourth Red):
OTS = (1×0.5 + 1+0+1) / (1 + 3) = 0.625 (prior + first three Reds)
No target leakage! Each sample only uses previous samples' targets.
1. Feature Quantization
def _quantize_features(self, X):
# For each feature
for feature_idx in range(n_features):
feature_values = X[:, feature_idx]
# Create borders using quantiles
if unique_values <= border_count:
borders = unique_values
else:
percentiles = linspace(0, 100, border_count+1)
borders = percentile(feature_values, percentiles)
# Assign bin indices
X_quantized[:, feature_idx] = digitize(feature_values, borders)2. Symmetric Tree Building
def _build_symmetric_tree(self, X_quantized, gradients):
splits = [] # Store splits for each level
partitions = [all_samples_mask] # Start with all data
for level in range(depth):
# Find ONE best split for ALL partitions
best_gain = -inf
for feature in features:
for threshold in thresholds:
# Calculate gain across ALL partitions
total_gain = sum(
gain_if_partition_split(p, feature, threshold)
for p in partitions
)
if total_gain > best_gain:
best = (feature, threshold)
# Apply best split to ALL partitions
splits.append(best)
partitions = [split_partition(p, best) for p in partitions]
# Calculate leaf values
leaf_values = [calculate_value(p, gradients) for p in partitions]
return {'splits': splits, 'leaf_values': leaf_values}3. Fast Symmetric Tree Prediction
def _predict_tree(self, tree, X_quantized):
n_samples = len(X_quantized)
leaf_indices = zeros(n_samples)
# Binary indexing for fast lookup
for level, split in enumerate(tree['splits']):
feature_idx = split['feature']
threshold = split['threshold']
# Samples going right: add to leaf index
goes_right = X_quantized[:, feature_idx] > threshold
remaining_depth = tree['depth'] - level - 1
leaf_indices += goes_right * (2 ** remaining_depth)
# Get predictions from leaf values
predictions = tree['leaf_values'][leaf_indices]
return predictions4. Leaf Value with L2 Regularization
def _calculate_leaf_value(self, gradients, indices):
gradient_sum = sum(gradients[indices])
count = sum(indices)
# CatBoost formula: shrinkage through L2 reg
value = -gradient_sum / (count + l2_leaf_reg)
return valueLet's work through a complete example: predicting house prices (regression).
House Data:
ID | Size(sqft) | Bedrooms | Age(years) | Price($k)
---|------------|----------|------------|----------
1 | 1200 | 2 | 10 | 180
2 | 1800 | 3 | 5 | 280
3 | 1500 | 3 | 15 | 220
4 | 2200 | 4 | 3 | 350
5 | 1000 | 2 | 20 | 150
6 | 2500 | 4 | 2 | 400
7 | 1400 | 2 | 12 | 200
8 | 2000 | 3 | 7 | 300
Size bins (border_count=2):
Bin 0: Size ≤ 1650 → [1200, 1500, 1000, 1400]
Bin 1: Size > 1650 → [1800, 2200, 2500, 2000]
Bedrooms bins:
Bin 0: Bedrooms ≤ 2.5 → [2, 2, 2]
Bin 1: Bedrooms > 2.5 → [3, 3, 4, 4, 3]
Age bins (border_count=2):
Bin 0: Age ≤ 8.5 → [10, 5, 3, 2, 7] (Hmm, boundary at 8.5)
Actually: Age ≤ 11 → [10, 5, 3, 2, 7, 12]
Age > 11 → [15, 20]
Let's say:
Bin 0: Age ≤ 8.5 → [5, 3, 2, 7]
Bin 1: Age > 8.5 → [10, 15, 20, 12]
Mean price: (180 + 280 + 220 + 350 + 150 + 400 + 200 + 300) / 8 = 260
base_score = 260
Initial predictions: [260, 260, 260, 260, 260, 260, 260, 260]
For regression (squared loss):
g = pred - y
ID | y | pred | gradient
---|-----|------|----------
1 | 180 | 260 | 80
2 | 280 | 260 | -20
3 | 220 | 260 | 40
4 | 350 | 260 | -90
5 | 150 | 260 | 110
6 | 400 | 260 | -140
7 | 200 | 260 | 60
8 | 300 | 260 | -40
Gradients: [80, -20, 40, -90, 110, -140, 60, -40]
LEVEL 0: Choose split for root
Try Size <= Bin 0 (Size ≤ 1650):
Left (Small houses): [1, 3, 5, 7]
Gradients: [80, 40, 110, 60]
G_L = 290, N_L = 4
Loss_L = -(290)² / (4 + 3) = -84100 / 7 = -12014.3
Right (Large houses): [2, 4, 6, 8]
Gradients: [-20, -90, -140, -40]
G_R = -290, N_R = 4
Loss_R = -(-290)² / (4 + 3) = -84100 / 7 = -12014.3
Parent:
G_P = 0, N_P = 8
Loss_P = 0² / (8 + 3) = 0
Gain = (Loss_L + Loss_R) - Loss_P = -24028.6 - 0 = -24028.6
(Note: We're looking at loss reduction, so more negative = better)
Best split: Size <= 1650
Current partitions:
├─ Partition 0 (Small): [1, 3, 5, 7]
└─ Partition 1 (Large): [2, 4, 6, 8]
LEVEL 1: Choose ONE split for BOTH partitions
Try Bedrooms <= Bin 0 (Bedrooms ≤ 2.5):
Partition 0 (Small houses):
Left (Small, ≤2 bed): [1, 5, 7] - All have 2 bedrooms
G = 80 + 110 + 60 = 250
Right (Small, >2 bed): [3] - Has 3 bedrooms
G = 40
Partition 1 (Large houses):
Left (Large, ≤2 bed): [] - None
Right (Large, >2 bed): [2, 4, 6, 8] - All have 3-4 bedrooms
G = -290
Calculate gain for this split...
Best split: Bedrooms <= 2.5
Final tree:
[Size <= 1650]
/ \
[Bedrooms <= 2.5] [Bedrooms <= 2.5]
/ \ / \
Leaf0 Leaf1 Leaf2 Leaf3
[1,5,7] [3] [] [2,4,6,8]
Leaf 0 (Small, ≤2 bed): [1, 5, 7]
G = 250, N = 3
value = -250 / (3 + 3) = -41.67
Leaf 1 (Small, >2 bed): [3]
G = 40, N = 1
value = -40 / (1 + 3) = -10.00
Leaf 2 (Large, ≤2 bed): []
value = 0 (empty leaf)
Leaf 3 (Large, >2 bed): [2, 4, 6, 8]
G = -290, N = 4
value = -(-290) / (4 + 3) = 41.43
Learning rate η = 0.05
House 1 (Leaf 0): 260 + 0.05 × (-41.67) = 260 - 2.08 = 257.92
House 3 (Leaf 1): 260 + 0.05 × (-10.00) = 260 - 0.50 = 259.50
House 2 (Leaf 3): 260 + 0.05 × 41.43 = 260 + 2.07 = 262.07
...
After this iteration:
- Small houses with ≤2 bed: predictions decrease (were overestimated)
- Large houses with >2 bed: predictions increase (were underestimated)
- Model is learning the pattern!
Iteration 2: Calculate new gradients from updated predictions
Iteration 3: Build another symmetric tree
...
Iteration 100: Final model
Final predictions after 100 trees:
House 1 (Small, 2 bed, old): Predicted ≈ 175 (actual 180) ✓
House 4 (Large, 4 bed, new): Predicted ≈ 355 (actual 350) ✓
House 5 (Smallest, 2 bed, old): Predicted ≈ 145 (actual 150) ✓
New house [1600 sqft, 3 bed, 8 years]:
1. Size 1600 ≤ 1650? YES → Partition 0
2. Bedrooms 3 > 2.5? YES → Leaf 1
3. Sum contributions from Leaf 1 across all trees
4. Final prediction ≈ 235k
Problem: Automatically categorize products from titles and features
Why CatBoost?
- Many categorical features (brand, seller, category)
- Handles text-derived features naturally
- Fast training for millions of products
- Excellent accuracy out-of-the-box
Features:
Text: product_title_words (categorical)
Categorical: brand, seller_id, existing_category
Numerical: price, weight, dimensions
Derived: brand_price_segment, title_length
Benefits:
- No need for extensive one-hot encoding
- Natural handling of rare brands/sellers
- Robust to new categories
- 95%+ categorization accuracy
Problem: Predict loan default risk
Why CatBoost?
- Handles missing values well
- Categorical features (occupation, location)
- Ordered boosting prevents overfitting on small datasets
- Regulatory compliance (explainable predictions)
Features:
Categorical: occupation, city, education, marital_status
Numerical: income, debt_to_income, credit_score, age
Derived: income_to_loan_ratio, employment_stability_score
Benefits:
- Better risk assessment (20-30% improvement over logistic regression)
- Handles rare occupations/locations without overfitting
- Feature importance for regulatory explanation
- Robust with default parameters
Problem: Predict total revenue from each customer
Why CatBoost?
- Mix of categorical and numerical features
- Long tail of customer behaviors
- Need for accurate predictions across segments
Features:
Categorical: acquisition_channel, first_product_category, location_tier
Numerical: days_since_signup, total_orders, avg_order_value
Behavioral: browsing_frequency, email_engagement, support_contacts
Benefits:
- Accurate CLV predictions enable targeted marketing
- Segment customers effectively
- 15-20% improvement over traditional methods
- Fast retraining with new data
Problem: Assist in disease diagnosis from symptoms and test results
Why CatBoost?
- Categorical symptoms (yes/no, severity levels)
- Numerical lab values
- Handles missing test results naturally
- High accuracy requirements
Features:
Categorical: symptoms (fever, cough, fatigue), medical_history
Numerical: lab_values (blood_pressure, glucose, white_cell_count)
Demographic: age, gender, bmi
Benefits:
- High diagnostic accuracy (comparable to specialists)
- Probability scores help prioritize cases
- Interpretable feature importance
- Robust to missing lab values
Problem: Predict if website visitor will convert
Why CatBoost?
- Many categorical features (device, browser, referrer)
- Session-based features
- Need for fast online predictions
- Handle cold-start (new visitors)
Features:
Categorical: traffic_source, device_type, browser, landing_page
Behavioral: pages_viewed, time_on_site, scroll_depth
Contextual: day_of_week, hour, season
Historical: previous_visits, email_subscriber
Benefits:
- Real-time conversion probability
- Personalized content recommendations
- 10-15% increase in conversion rate
- Fast prediction (< 1ms per user)
class CatBoost:
def __init__(self, n_estimators=100, learning_rate=0.03,
depth=6, l2_leaf_reg=3.0, ...):
# Key parameters
self.depth = depth # Tree depth (controls 2^depth leaves)
self.learning_rate = learning_rate # Shrinkage
self.l2_leaf_reg = l2_leaf_reg # Regularization strength
# ...
def fit(self, X, y):
# 1. Quantize features
# 2. Initialize predictions
# 3. Train symmetric trees sequentially
def predict(self, X):
# 1. Apply quantization
# 2. Fast prediction using binary indexing
# 3. Convert to probabilities if classification1. Feature Quantization
def _quantize_features(self, X):
"""
Convert continuous features to discrete bins
Why: Faster split evaluation and more robust
- Original: Try every unique value
- Quantized: Try only bin boundaries
- Adds regularization through discretization
Example:
Prices: [100, 150, 180, 220, 250, 300]
With border_count=3:
Bin 0 (≤165), Bin 1 (165-235), Bin 2 (>235)
"""2. Symmetric Tree Building
def _build_symmetric_tree(self, X_quantized, gradients):
"""
Build tree where all nodes at same level use same split
Why symmetric trees?
- Natural regularization (simpler structure)
- Faster prediction (binary indexing)
- Less prone to overfitting
- Easier to parallelize
Algorithm:
- For each level (0 to depth-1):
* Find ONE best split for ALL current partitions
* Apply it to ALL partitions
* Double the number of partitions
- Result: 2^depth leaves with symmetric structure
"""3. Fast Prediction with Binary Indexing
def _predict_tree(self, tree, X_quantized):
"""
Fast prediction using binary representation of tree path
Why fast?
- Traditional tree: Follow path, could be O(depth × branches)
- Symmetric tree: Just compute leaf index, O(depth)
Algorithm:
1. Start with leaf_index = 0
2. For each level's split:
- If goes RIGHT: add 2^(remaining_depth)
- If goes LEFT: add 0
3. Return leaf_value[leaf_index]
Example (depth=3):
Path: R-L-R
Index: 0 + 4 + 0 + 1 = 5
Return: leaf_value[5]
"""4. Leaf Value with Strong Regularization
def _calculate_leaf_value(self, gradients, indices):
"""
Calculate optimal leaf value with L2 regularization
Formula: value = -sum(gradients) / (count + l2_leaf_reg)
Why L2 in denominator?
- Shrinks leaf values toward zero
- More shrinkage for small leaves (low count)
- Less shrinkage for large leaves (high count)
- Prevents overfitting to small groups
Example:
Leaf with 100 samples, sum(g)=-50, λ=3:
value = 50 / (100 + 3) = 0.485
Leaf with 5 samples, sum(g)=-50, λ=3:
value = 50 / (5 + 3) = 6.25 (less shrinkage needed)
CatBoost default λ=3.0 is higher than XGBoost's 1.0!
"""Tree Structure:
depth=6 # Tree depth (2^6 = 64 leaves)
# Controls model complexity
# Typical: 4-10
min_data_in_leaf=1 # Min samples per leaf
# CatBoost trusts regularization, uses 1Learning:
learning_rate=0.03 # Shrinkage (lower than XGBoost/LightGBM)
# CatBoost uses conservative default
# Typical: 0.01-0.1
n_estimators=100 # Number of trees
# More trees with lower learning rateRegularization:
l2_leaf_reg=3.0 # L2 regularization strength
# Higher than XGBoost default (1.0)
# Strong regularization prevents overfitting
# Typical: 1-10Speed vs Accuracy:
border_count=128 # Number of feature bins
# Higher = more accurate, slower
# Typical: 32, 64, 128, 254Randomness:
random_strength=1.0 # Randomization in split selection
# Adds small random value to gain
# Helps with generalization
# Typical: 0-2Start with defaults:
model = CatBoost(
n_estimators=100,
learning_rate=0.03,
depth=6,
l2_leaf_reg=3.0
)
# CatBoost has great defaults! Often works well as-is.If underfitting (train and test loss both high):
# Increase model complexity
model = CatBoost(
n_estimators=200, # More trees
depth=8, # Deeper trees
learning_rate=0.05, # Slightly higher rate
l2_leaf_reg=1.0 # Less regularization
)If overfitting (train loss low, test loss high):
# Increase regularization
model = CatBoost(
n_estimators=100,
depth=4, # Shallower trees
learning_rate=0.03,
l2_leaf_reg=10.0, # More regularization
random_strength=2.0 # More randomness
)Regression:
# RMSE (Root Mean Squared Error)
rmse = -model.score(X_test, y_test) # Note: score returns negative RMSE
print(f"RMSE: {rmse:.2f}")
# Mean Absolute Error
predictions = model.predict(X_test)
mae = np.mean(np.abs(y_test - predictions))
print(f"MAE: {mae:.2f}")
# R² Score
ss_total = np.sum((y_test - np.mean(y_test)) ** 2)
ss_residual = np.sum((y_test - predictions) ** 2)
r2 = 1 - (ss_residual / ss_total)
print(f"R²: {r2:.4f}")Classification:
# Accuracy
accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy:.2%}")
# Confusion Matrix and Metrics
predictions = model.predict(X_test)
predicted_classes = (predictions >= 0.5).astype(int)
TP = np.sum((predicted_classes == 1) & (y_test == 1))
FP = np.sum((predicted_classes == 1) & (y_test == 0))
FN = np.sum((predicted_classes == 0) & (y_test == 1))
TN = np.sum((predicted_classes == 0) & (y_test == 0))
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)
print(f"Precision: {precision:.2%}")
print(f"Recall: {recall:.2%}")
print(f"F1 Score: {f1:.4f}")
# ROC-AUC
# (Would need to implement or use sklearn for full ROC curve)# Train model
model.fit(X_train, y_train)
# Get importance
importance = model.get_feature_importance('split')
# Display
feature_names = ['size', 'bedrooms', 'age', 'location']
for name, imp in sorted(zip(feature_names, importance),
key=lambda x: x[1], reverse=True):
bar = '█' * int(imp * 50)
print(f"{name:15s}: {imp:.4f} {bar}")
# Output:
# size : 0.4821 ████████████████████████
# bedrooms : 0.3012 ███████████████
# age : 0.1567 ████████
# location : 0.0600 ███# Train with validation set
model.fit(
X_train, y_train,
eval_set=[(X_val, y_val)],
verbose=False
)
# Plot learning curves
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 6))
plt.plot(model.train_scores, label='Train')
plt.plot(model.val_scores, label='Validation')
plt.xlabel('Iteration')
plt.ylabel('Loss')
plt.title('Learning Curves')
plt.legend()
plt.grid(True)
plt.show()
# Interpret:
# - Train and val decreasing: Model learning well
# - Val starts increasing: Overfitting, use early stopping
# - Val plateaus: Model converged, can stop early# Manual K-Fold Cross-Validation
from sklearn.model_selection import KFold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = []
for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
X_train_fold, X_val_fold = X[train_idx], X[val_idx]
y_train_fold, y_val_fold = y[train_idx], y[val_idx]
model = CatBoost(n_estimators=100, learning_rate=0.03, depth=6)
model.fit(X_train_fold, y_train_fold)
score = model.score(X_val_fold, y_val_fold)
scores.append(score)
print(f"Fold {fold+1}: {score:.4f}")
print(f"\nMean CV Score: {np.mean(scores):.4f} ± {np.std(scores):.4f}")Signs of Overfitting:
train_score = model.score(X_train, y_train) # 0.95
test_score = model.score(X_test, y_test) # 0.72
# Large gap = overfitting!Solutions:
- Use Early Stopping:
model.fit(
X_train, y_train,
eval_set=[(X_val, y_val)],
early_stopping_rounds=20 # Stop if no improvement for 20 rounds
)- Reduce Tree Depth:
model = CatBoost(depth=4) # Was 6, now shallower- Increase L2 Regularization:
model = CatBoost(l2_leaf_reg=10.0) # Was 3.0, now stronger- Lower Learning Rate with More Trees:
model = CatBoost(
n_estimators=300, # More trees
learning_rate=0.01 # Lower rate
)- Add Randomness:
model = CatBoost(random_strength=2.0) # More randomization| Feature | XGBoost | LightGBM | CatBoost |
|---|---|---|---|
| Tree Growth | Level-wise | Leaf-wise | Level-wise (symmetric) |
| Default LR | 0.3 | 0.1 | 0.03 |
| Categorical Handling | Manual encoding | Manual encoding | Native (automatic) |
| Speed (Large Data) | Medium | Fastest | Fast |
| Overfitting Risk | Medium | Higher | Lower |
| Default Performance | Good | Good | Best |
| Tree Structure | Asymmetric | Asymmetric | Symmetric |
| Best For | Competitions | Speed & large data | Categoricals & robustness |
Use XGBoost when:
- Industry standard needed
- Extensive documentation/resources needed
- Medium-sized datasets
- Have time for hyperparameter tuning
- Need ecosystem support (wide community)
Use LightGBM when:
- Speed is critical
- Very large datasets (>1M samples)
- Memory is limited
- Numerical features dominate
- Need GPU acceleration
Use CatBoost when:
- Many categorical features ← Best choice!
- Want great results with default parameters
- Overfitting is a concern
- Need robust, production-ready model
- Limited time for tuning
Typical benchmark results:
Numerical features only:
LightGBM ≈ XGBoost ≈ CatBoost
Many categorical features:
CatBoost > LightGBM > XGBoost
Small datasets (<10K):
CatBoost ≈ XGBoost > LightGBM
Large datasets (>100K):
LightGBM ≥ CatBoost > XGBoost
Default parameters:
CatBoost > LightGBM > XGBoost
(CatBoost has best defaults!)
Dataset: 100K samples, 50 features
Training Time:
├── XGBoost: 45 seconds
├── LightGBM: 12 seconds ← Fastest!
└── CatBoost: 30 seconds
Prediction Time (1000 samples):
├── XGBoost: 15 ms
├── LightGBM: 8 ms
└── CatBoost: 5 ms ← Fastest! (symmetric trees)
Why CatBoost prediction is fast?
- Symmetric trees → O(depth) instead of O(depth × branches)
- Binary indexing → direct lookup
- No need for tree traversal
-
CatBoost = Robustness + Categorical Intelligence
- Symmetric trees → Natural regularization
- Ordered boosting → Prevents target leakage
- Native categorical handling → No manual encoding needed
- Great defaults → Works well out-of-the-box
-
Main Innovations
- Symmetric Trees: All nodes at level use same split → simpler, faster
- Ordered Boosting: Prevents prediction shift → more robust
- Ordered Target Statistics: Smart categorical encoding → no leakage
- Strong Regularization: High default L2 (3.0) → less overfitting
-
Best Practices
# Start here (usually works great!) model = CatBoost( n_estimators=100, learning_rate=0.03, depth=6, l2_leaf_reg=3.0 ) # If underfitting model = CatBoost( n_estimators=200, depth=8, learning_rate=0.05, l2_leaf_reg=1.0 ) # If overfitting model = CatBoost( depth=4, l2_leaf_reg=10.0, random_strength=2.0 )
-
When to Use CatBoost
- ✅ Many categorical features (best choice!)
- ✅ Want good results with minimal tuning
- ✅ Concerned about overfitting
- ✅ Need robust production model
- ✅ Small to medium datasets
- ❌ Very large datasets where speed is critical (use LightGBM)
XGBoost: "The Industry Standard"
+ Mature, well-documented
+ Good balance of speed and accuracy
- Needs more tuning
- Manual categorical handling
LightGBM: "The Speed Champion"
+ Fastest on large datasets
+ Memory efficient
+ Great for numerical features
- Easier to overfit
- Needs careful tuning
CatBoost: "The Robust Expert"
+ Best with categorical features
+ Great default parameters
+ Less prone to overfitting
+ Fastest prediction (symmetric trees)
- Slower training than LightGBM
- Newer, smaller community
Do you have categorical features?
├─ YES → Use CatBoost
└─ NO → Is speed critical?
├─ YES → Use LightGBM
└─ NO → Use XGBoost or CatBoost
Is dataset small (<10K samples)?
├─ YES → Use CatBoost (more robust)
└─ NO → Is it huge (>1M samples)?
├─ YES → Use LightGBM (fastest)
└─ NO → Use CatBoost (best defaults)
Limited time for tuning?
└─ Use CatBoost (best out-of-the-box)
- Run the examples in the
.pyfile - Compare with your data - try defaults first!
- Add categorical features - see CatBoost shine
- Monitor for overfitting - use validation set
- Compare with XGBoost/LightGBM - see the differences
- Study symmetric trees - understand the structure
- CatBoost Documentation: https://catboost.ai/
- Paper: "CatBoost: unbiased boosting with categorical features" (NeurIPS 2018)
- GitHub: https://github.com/catboost/catboost
- Tutorial: https://catboost.ai/docs/concepts/tutorials.html
- Symmetric (oblivious) decision trees
- Ordered boosting and prediction shift
- Ordered target statistics for categorical features
- Comparison with XGBoost and LightGBM
- Handling of missing values
- XGBoost (main competitor, asymmetric trees)
- LightGBM (main competitor, leaf-wise growth)
- Gradient Boosting (foundation algorithm)
- Random Forests (alternative ensemble method)
- GPU acceleration in CatBoost
- Distributed training
- Custom loss functions
- Text features handling
- Embeddings for categorical features
Remember: CatBoost is robust and smart! It's especially powerful when you have categorical features and want excellent results without extensive tuning. Happy learning! 🚀
This guide is part of the "ML Algorithms from Scratch" series. For more algorithms, check out the repository!