Skip to content

NaturalBoost

NaturalBoost is distributional regression via boosting: each parameter of F(y | x) (loc, scale, …) is its own ensemble, stepped with natural gradient. That is the GAMLSS / NGBoost model class, with a GPU histogram-tree path.

For a structural formula that is not a distribution, use FormulaBoost (varying-coefficient). For right-censored Weibull survival, use WeibullAFT. Shared engine: How it works.

Why Uncertainty Matters

Traditional gradient boosting gives you a single number: "the price will be $100". But in reality, you might want to know:

  • How confident is the model?
  • What's the range of likely values?
  • What's the probability of exceeding a threshold?

NaturalBoost answers these questions by predicting distribution parameters (e.g., mean and variance for a Normal distribution).

Quick Start

import numpy as np
import openboost as ob

# Train probabilistic model
model = ob.NaturalBoostNormal(n_trees=100, max_depth=4)
model.fit(X_train, y_train)

# Point prediction (mean)
mean = model.predict(X_test)

# 90% prediction interval
lower, upper = model.predict_interval(X_test, alpha=0.1)

# Check coverage
coverage = np.mean((y_test >= lower) & (y_test <= upper))
print(f"Coverage: {coverage:.1%}")  # Should be ~90%

Available Models

Model Distribution Use Case
NaturalBoostNormal Gaussian General uncertainty
NaturalBoostLogNormal Log-Normal Positive skewed (prices)
NaturalBoostGamma Gamma Positive continuous
NaturalBoostPoisson Poisson Count data
NaturalBoostStudentT Student-t Heavy tails, outliers
NaturalBoostTweedie Tweedie Insurance claims (Kaggle!)
NaturalBoostNegBin Negative Binomial Sales forecasting (Kaggle!)

Distribution Output

For full control, use predict_distribution():

output = model.predict_distribution(X_test)

# Access distribution parameters
mean = output.mean()
std = output.std()
variance = output.variance()

# Prediction intervals
lower, upper = output.interval(alpha=0.1)  # 90% interval

# Negative log-likelihood
nll = output.nll(y_test)
print(f"Mean NLL: {np.mean(nll):.4f}")

Monte Carlo Sampling

Sample from the predicted distribution for downstream analysis:

# Draw samples
samples = model.sample(X_test, n_samples=1000)  # Shape: (1000, n_test)

# Risk analysis
threshold = 10.0
prob_exceed = np.mean(samples > threshold, axis=0)  # P(Y > 10)

# Quantile estimation
q90 = np.percentile(samples, 90, axis=0)

vs NGBoost

On GPU (Modal A100, 90K rows, heteroscedastic Normal) NaturalBoost fits in 2.21s against NGBoost's 2716s, with NLL tied (2.108 vs 2.102). NGBoost has no GPU implementation. This is one configuration from an early benchmark run, not a settled result.

On the NGBoost-paper UCI suite (20 paired splits, same budget) OpenBoost is tied-or-better on every dataset that completed; significant NLL wins on kin8nm, protein, and california; no significant loss.

On CPU the two libraries are ~parity (0.8–1.3×). The speed claim is the GPU tree path, not a faster CPU NGBoost clone.

Full tables, caveats, and reproduce commands: Benchmarks.

Best Practices

  1. Use shallower trees (max_depth=3-4) - better for uncertainty estimation
  2. Train longer - NaturalBoost learns 2+ parameters per sample
  3. Evaluate with NLL - not just RMSE
model = ob.NaturalBoostNormal(
    n_trees=500,      # More trees
    max_depth=4,      # Shallower
    learning_rate=0.05,  # Lower LR
)