Skip to content

_3_ridge_regression.md

Latest commit

 

History

History
695 lines (523 loc) · 19.7 KB

_3_ridge_regression.md

File metadata and controls

695 lines (523 loc) · 19.7 KB
Raw

Ridge Regression from Scratch: A Comprehensive Guide

Welcome to the world of Ridge Regression! 🎯 In this comprehensive guide, we'll explore how to predict outcomes using regularized linear regression. Think of it as an improved version of linear regression that prevents overfitting and handles multicollinearity like a pro!

Table of Contents

  1. What is Ridge Regression?
  2. Why Do We Need Ridge Regression?
  3. The Mathematical Foundation
  4. Implementation Details
  5. Step-by-Step Example
  6. Real-World Applications
  7. Understanding the Code
  8. Ridge vs Multiple Regression

What is Ridge Regression?

Ridge Regression is a regularized version of Multiple Linear Regression that adds a penalty term to prevent overfitting and reduce the impact of multicollinearity. It's one of the most important techniques in machine learning for building robust, generalizable models.

Real-world analogy: Imagine you're a teacher grading students based on test scores, homework, and participation. If you weight one factor too heavily (like giving 95% weight to just test scores), you might miss important patterns. Ridge regression is like ensuring all factors contribute reasonably, preventing any single factor from dominating the prediction!

The Mathematical Equation

The prediction formula remains the same as multiple regression:

y = b₀ + b₁x₁ + b₂x₂ + b₃x₃ + ... + bₙxₙ

But the way we calculate coefficients changes:

Multiple Regression (No Regularization):

θ = (XᵀX)⁻¹Xᵀy

Ridge Regression (With L2 Regularization):

θ = (XᵀX + λI)⁻¹Xᵀy

Where:

  • λ (lambda/alpha) = regularization parameter (strength of penalty)
  • I = identity matrix (diagonal matrix of ones)
  • All other terms same as before

Why Do We Need Ridge Regression?

Problem 1: Overfitting

What is overfitting? When a model learns the training data too well - including noise and random fluctuations - it performs poorly on new, unseen data.

Example:

# Training data: 10 samples, 20 features
# Model learns: "Feature 17 is THE most important!"
# Reality: Feature 17 just happened to correlate by chance

# Result: Great training accuracy, poor test accuracy

How Ridge helps: By penalizing large coefficients, Ridge prevents the model from relying too heavily on any single feature.

Problem 2: Multicollinearity

What is multicollinearity? When features are highly correlated with each other, making it hard to determine their individual effects.

Example:

Feature 1: House square footage = 2000
Feature 2: House area in meters = 185.8 (almost the same info!)

Problem: Model doesn't know which feature is truly important
Result: Unstable, unreliable coefficients

How Ridge helps: The regularization term stabilizes the coefficient estimates, even when features are correlated.

Problem 3: High-Dimensional Data

When you have many features relative to the number of samples:

  • Matrix (XᵀX) becomes singular or nearly singular
  • Inverse calculation becomes unstable or impossible
  • Coefficients become unrealistically large

How Ridge helps: Adding λI to (XᵀX) ensures the matrix is always invertible!

The Mathematical Foundation

Understanding the Regularization Term

Cost Function:

Multiple Regression minimizes:

Cost = Σ(y - ŷ)²

Ridge Regression minimizes:

Cost = Σ(y - ŷ)² + λΣ(βⱼ)²

Breaking it down:

  1. Σ(y - ŷ)² = Prediction error (we want this small)
  2. λΣ(βⱼ)² = Penalty for large coefficients (L2 regularization)
  3. λ = Controls the trade-off between fitting data and keeping coefficients small

The Lambda (α) Parameter

The regularization parameter λ (also called alpha) controls the strength of regularization:

Lambda Value Effect When to Use
λ = 0 No regularization (same as Multiple Regression) Data is clean, no multicollinearity
λ = 0.01 - 0.1 Light regularization Mild overfitting concerns
λ = 1.0 Moderate regularization Balanced approach (often a good start)
λ = 10 - 100 Strong regularization Severe overfitting or multicollinearity
λ → ∞ All coefficients → 0 Model predicts only the mean

The Normal Equation with Regularization

Starting from:

θ = (XᵀX)⁻¹Xᵀy

We add λI to make it more stable:

θ = (XᵀX + λI)⁻¹Xᵀy

Why this works:

  1. XᵀX is positive semi-definite
  2. λI adds positive values to the diagonal
  3. XᵀX + λI is always positive definite (invertible!)
  4. Larger λ → smaller coefficients

Important Note: We typically don't regularize the intercept term, so the identity matrix has 0 in the first position.

Implementation Details

Our implementation includes the following key components:

Class Structure

class RidgeRegression:
    def __init__(self, alpha=1.0):
        self.alpha = alpha  # Regularization strength
        self.coefficients = None
        self.intercept = None

Core Methods

  1. __init__(alpha=1.0) - Initialize model

    • Set regularization strength (alpha/lambda)
    • Default alpha = 1.0 (moderate regularization)

  2. fit(X, y) - Train the model

    • Adds bias term (column of ones)
    • Creates identity matrix with 0 for intercept position
    • Calculates coefficients using regularized Normal Equation
    • Stores intercept and feature coefficients separately

  3. predict(X) - Make predictions

    • Adds bias term to new data
    • Applies the linear equation with learned coefficients
    • Returns predicted values

  4. get_coefficients() - Get model parameters

    • Returns intercept, coefficients, and alpha
    • Useful for understanding feature importance

  5. score(X, y) - Calculate R² score

    • Measures how well the model fits the data
    • Returns value between 0 and 1 (1 = perfect fit)

Step-by-Step Example

Let's walk through a complete example predicting house prices with Ridge Regression:

The Data

import numpy as np

# Features: [square_feet, bedrooms, age_of_house]
X_train = np.array([
    [1500, 3, 10],
    [2000, 4, 5],
    [1200, 2, 15],
    [1800, 3, 8],
    [2500, 5, 2],
    [1700, 3, 12],
    [2200, 4, 6],
    [1400, 2, 20]
])

# Target: house prices in dollars
y_train = np.array([300000, 400000, 250000, 350000, 500000, 
                     320000, 420000, 280000])

Comparing Different Alpha Values

# Try different regularization strengths
alphas = [0.0, 0.1, 1.0, 10.0, 100.0]

for alpha in alphas:
    model = RidgeRegression(alpha=alpha)
    model.fit(X_train, y_train)
    
    coeffs = model.get_coefficients()
    print(f"\nAlpha = {alpha}")
    print(f"Coefficients: {coeffs['coefficients']}")

Expected observation:

  • As alpha increases, coefficient magnitudes decrease
  • Alpha = 0 gives same results as Multiple Regression
  • Very large alpha shrinks coefficients toward zero

Training with Optimal Alpha

# Use cross-validation or domain knowledge to choose alpha
model = RidgeRegression(alpha=1.0)
model.fit(X_train, y_train)

# Make predictions
X_test = np.array([
    [1600, 3, 7],   # 1600 sq ft, 3 bedrooms, 7 years old
    [2200, 4, 3]    # 2200 sq ft, 4 bedrooms, 3 years old
])

predictions = model.predict(X_test)
print("Predicted prices:", predictions)

Interpreting Results

coeffs = model.get_coefficients()
print(f"Intercept: ${coeffs['intercept']:.2f}")
print(f"Square Feet Coefficient: ${coeffs['coefficients'][0]:.2f}")
print(f"Bedrooms Coefficient: ${coeffs['coefficients'][1]:.2f}")
print(f"Age Coefficient: ${coeffs['coefficients'][2]:.2f}")

What the coefficients mean (with regularization):

  • Values are typically smaller than unregularized regression
  • More stable and generalizable
  • Better represent true feature importance
  • Less affected by noise and multicollinearity

Real-World Applications

1. Financial Modeling

Predicting stock returns based on multiple indicators:

  • Multiple financial ratios (often correlated)
  • Ridge handles multicollinearity between ratios
  • Prevents overfitting to historical patterns
  • More stable predictions

2. Medical Research

Predicting patient outcomes:

  • Many biomarkers and health indicators
  • Often have many features, fewer patients
  • Ridge prevents overfitting to training data
  • Reliable predictions for new patients

3. Real Estate Valuation

Predicting property prices:

  • Multiple features (size, location, amenities)
  • Some features highly correlated
  • Ridge provides stable price estimates
  • Generalizes well to new properties

4. Marketing Analytics

Predicting customer behavior:

  • Multiple marketing channels (TV, radio, online, social media)
  • Channels often correlated in campaigns
  • Ridge handles multicollinearity
  • Identifies true channel effectiveness

5. Climate Modeling

Predicting environmental variables:

  • Many correlated measurements
  • Complex feature interactions
  • Ridge provides stable predictions
  • Prevents overfitting to historical noise

Understanding the Code

Let's break down the key parts of our implementation:

1. Creating the Identity Matrix

identity = np.eye(X_with_bias.shape[1])
identity[0, 0] = 0  # Don't penalize the intercept

Why?

  • We want to regularize feature coefficients, not the intercept
  • The intercept represents the base value when all features are 0
  • Penalizing it would bias our predictions

Example:

For 3 features + intercept:
identity = [[0, 0, 0, 0],
            [0, 1, 0, 0],
            [0, 0, 1, 0],
            [0, 0, 0, 1]]

2. Regularized Normal Equation

regularization_term = self.alpha * identity
self.coefficients = np.linalg.inv(X_with_bias.T @ X_with_bias + regularization_term) @ X_with_bias.T @ y

Step-by-step:

  1. Create regularization term: λI
  2. Add to XᵀX: (XᵀX + λI)
  3. Invert the matrix: (XᵀX + λI)⁻¹
  4. Multiply by Xᵀy
  5. Result: Regularized coefficients!

What it does:

  • Adds λ to the diagonal of XᵀX
  • Makes the matrix better conditioned
  • Shrinks coefficients toward zero
  • More stable inversion

3. Effect of Alpha

# Small alpha (e.g., 0.1): Light regularization
# Coefficients: [150.5, 20000.3, -5000.8]

# Large alpha (e.g., 100): Strong regularization  
# Coefficients: [80.2, 5000.1, -1200.4]

Pattern:

  • Larger alpha → smaller coefficients
  • Smaller alpha → closer to unregularized solution
  • Find balance through cross-validation

4. When Does Ridge Help Most?

Ridge Regression provides the most benefit when:

  1. Many features relative to samples

    # Problematic scenario
n_samples = 100
n_features = 80  # Almost as many features as samples!

# Ridge to the rescue!
model = RidgeRegression(alpha=1.0)

  2. Features are correlated

    # High correlation between features
correlation_matrix = np.corrcoef(X.T)
# If many values > 0.8, Ridge helps!

  3. Overfitting is observed

    # Signs of overfitting
train_r2 = 0.95  # Very high
test_r2 = 0.60   # Much lower

# Ridge can help close this gap

Ridge vs Multiple Regression

Side-by-Side Comparison

Aspect Multiple Regression Ridge Regression
Formula θ = (XᵀX)⁻¹Xᵀy θ = (XᵀX + λI)⁻¹Xᵀy
Regularization None L2 (sum of squared coefficients)
Coefficient Size Can be very large Shrunk toward zero
Multicollinearity Problems with correlated features Handles it well
Overfitting Prone to overfitting Reduces overfitting
Bias-Variance Low bias, high variance Slightly higher bias, lower variance
Interpretability Highly interpretable Still interpretable
When to Use Clean data, few features Many/correlated features

Practical Comparison

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

# Load dataset
data = load_diabetes()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Multiple Regression
model_ols = MultipleRegression()
model_ols.fit(X_train, y_train)
train_r2_ols = model_ols.score(X_train, y_train)
test_r2_ols = model_ols.score(X_test, y_test)

# Ridge Regression
model_ridge = RidgeRegression(alpha=1.0)
model_ridge.fit(X_train, y_train)
train_r2_ridge = model_ridge.score(X_train, y_train)
test_r2_ridge = model_ridge.score(X_test, y_test)

print("Multiple Regression:")
print(f"  Train R²: {train_r2_ols:.4f}")
print(f"  Test R²:  {test_r2_ols:.4f}")
print(f"  Gap:      {train_r2_ols - test_r2_ols:.4f}")

print("\nRidge Regression:")
print(f"  Train R²: {train_r2_ridge:.4f}")
print(f"  Test R²:  {test_r2_ridge:.4f}")
print(f"  Gap:      {train_r2_ridge - test_r2_ridge:.4f}")

Typical Results:

  • Ridge may have slightly lower training R²
  • Ridge often has higher (or similar) test R²
  • Ridge has smaller train-test gap (better generalization)

Choosing the Right Alpha

Methods to Select Alpha

  1. Cross-Validation (Best Practice)

    from sklearn.model_selection import cross_val_score

alphas = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
for alpha in alphas:
    model = RidgeRegression(alpha=alpha)
    # Perform k-fold cross-validation
    # Choose alpha with best average score

  2. Grid Search

    # Try many alpha values
# Evaluate on validation set
# Pick the one with best performance

  3. Domain Knowledge

    • Start with alpha = 1.0 as baseline
    • If overfitting: increase alpha
    • If underfitting: decrease alpha

Alpha Selection Guidelines

Scenario Suggested Alpha Range
Clean data, few features 0.01 - 0.1
Moderate complexity 0.1 - 10
Many features, small dataset 1.0 - 100
Severe multicollinearity 10 - 1000
Just want to try Ridge Start with 1.0

Key Concepts to Remember

1. Bias-Variance Tradeoff

  • Ridge increases bias slightly
  • Ridge decreases variance significantly
  • Net result: Better generalization

2. Feature Scaling

Ridge regression is sensitive to feature scales! Always normalize/standardize:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

3. Ridge Never Sets Coefficients to Zero

  • Ridge shrinks coefficients toward zero
  • But never makes them exactly zero
  • All features remain in the model
  • For feature selection, use Lasso Regression instead

4. Computational Advantages

  • Closed-form solution (no iteration needed)
  • Fast to train
  • Always finds global optimum
  • No hyperparameter tuning except alpha

Complete Usage Example

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load diabetes dataset (10 features)
data = load_diabetes()
X, y = data.data, data.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Important: Standardize features for Ridge
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Try different alpha values
print("Alpha Selection:\n")
for alpha in [0.01, 0.1, 1.0, 10.0, 100.0]:
    model = RidgeRegression(alpha=alpha)
    model.fit(X_train_scaled, y_train)
    
    train_r2 = model.score(X_train_scaled, y_train)
    test_r2 = model.score(X_test_scaled, y_test)
    
    print(f"Alpha = {alpha:6.2f} | Train R²: {train_r2:.4f} | Test R²: {test_r2:.4f}")

# Train with best alpha (determined from above)
print("\n" + "="*50)
print("Final Model with Alpha = 1.0")
print("="*50)

model = RidgeRegression(alpha=1.0)
model.fit(X_train_scaled, y_train)

# Make predictions
y_pred = model.predict(X_test_scaled)

# Evaluate model
r2 = model.score(X_test_scaled, y_test)
print(f"\nR² Score: {r2:.4f}")

# Examine coefficients
coeffs = model.get_coefficients()
print(f"\nIntercept: {coeffs['intercept']:.2f}")
print("\nFeature Coefficients:")
for i, coef in enumerate(coeffs['coefficients'], 1):
    print(f"  Feature {i}: {coef:.4f}")

# Compare coefficient magnitudes
coef_magnitude = np.linalg.norm(coeffs['coefficients'])
print(f"\nCoefficient L2 Norm: {coef_magnitude:.4f}")

Visualizing Ridge Regression

Here's how to visualize the effect of regularization:

import numpy as np
import matplotlib.pyplot as plt

# Generate coefficients for different alpha values
alphas = np.logspace(-2, 3, 50)  # 0.01 to 1000
coefficients = []

for alpha in alphas:
    model = RidgeRegression(alpha=alpha)
    model.fit(X_train_scaled, y_train)
    coefficients.append(model.get_coefficients()['coefficients'])

coefficients = np.array(coefficients)

# Plot coefficient paths
plt.figure(figsize=(12, 6))
for i in range(coefficients.shape[1]):
    plt.plot(alphas, coefficients[:, i], label=f'Feature {i+1}')

plt.xscale('log')
plt.xlabel('Alpha (Regularization Strength)', fontsize=12)
plt.ylabel('Coefficient Value', fontsize=12)
plt.title('Ridge Regression: Coefficient Paths', fontsize=14)
plt.axhline(y=0, color='black', linestyle='--', linewidth=0.5)
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

What you'll see:

  • As alpha increases, all coefficients shrink toward zero
  • Some coefficients shrink faster than others
  • No coefficient reaches exactly zero
  • Trade-off between fitting data and keeping coefficients small

Assumptions and Limitations

Assumptions

Ridge Regression assumes:

  • Linear relationship between features and target
  • Errors are normally distributed
  • Constant variance of errors (homoscedasticity)
  • Features are somewhat independent (though handles multicollinearity better than OLS)

Limitations

  1. Feature Scaling Required

    • Must standardize features
    • Different scales → unequal penalization

  2. Doesn't Perform Feature Selection

    • All features remain in model
    • Coefficients shrink but never reach zero
    • Use Lasso for feature selection

  3. Alpha Selection Needed

    • Requires cross-validation
    • Extra computational step
    • Results depend on alpha choice

  4. Still Assumes Linearity

    • Can't capture non-linear relationships
    • Use polynomial features or non-linear models for that

Conclusion

Ridge Regression is a powerful enhancement to linear regression that provides:

  • Robustness against overfitting
  • Stability in the presence of multicollinearity
  • Better generalization to new data
  • Computational efficiency with closed-form solution

By adding a simple regularization term, we get a model that is more reliable and practical for real-world applications! 🎯

When to Use Ridge Regression:

  • ✅ Many features relative to samples
  • ✅ Features are correlated
  • ✅ Overfitting is a concern
  • ✅ Want stable, interpretable coefficients
  • ✅ Need all features in the model

When to Use Something Else:

  • ❌ Need feature selection → Use Lasso
  • ❌ Non-linear relationships → Use polynomial features or tree models
  • ❌ Very large datasets → Consider gradient descent methods

Next Steps:

  • Try with your own data
  • Experiment with different alpha values
  • Compare with Multiple Regression and Lasso
  • Visualize coefficient paths
  • Learn about cross-validation for alpha selection

Happy coding! 💻📊