Stacker¶
Ensemble learning combining multiple regression models into a single regression pipeline
Shahar Azulay, Ariel Hanemann
There are three so-called “meta-algorithms” / approaches to combine several machine learning techniques into one predictive model in order to:
- decrease the variance (bagging)
- decrease the bias (boosting)
- or improving the predictive force (stacking).
Stacking is an ensemble learning method in which one applys several models to your original data at the first stage. Each of the models are trained to fit and best estimate the data in a predictive manner.
At the second stage, the models are not combined in a standard way (e.g. averaging their outputs) but rather meta-level model is introduced to estimate the input together with outputs of every model to estimate the weight each model should get or, in other words, determine which models perform better than others on the certain types of input data.
[1] high-level description of the stacking ensemble
Stacking is tricky and should be performed with care. Without a careful implementation stacking can be easily drawn into a very overfitted model, that fails to generalize well on the input data.
The Stacker module allows for easy use of the stacking apporoach in an sklearn-based environment.
First time example:
>>> from sklearn.decomposition import PCA
>>> from sklearn.pipeline import Pipeline
>>> from sklearn.tree import DecisionTreeRegressor
>>> from sklearn.linear_model import LinearRegression
>>> import stacker
>>>
>>> from sklearn.datasets import load_boston
>>> X, y = load_boston(return_X_y=True)
>>>
>>> tr, te = self._default_cv_fun().split(X).next()
>>> X_train, X_test = X[tr], X[te]
>>>
>>> my_pipe = Pipeline([
('pca', PCA()),
('dtr', DecisionTreeRegressor(random_state=1))])
>>> my_regr = LinearRegression()
>>> my_meta_regr = LinearRegression()
>>>
>>> stck = stacker.Stacker(
first_level_preds=[my_pipe, my_regr],
stacker_pred=my_meta_regr,
n_jobs=-1)
>>>
>>> stck.fit(X[tr], y[tr])
>>> y_hat = stck.predict(X[te])
Contents¶
The Stacking Ensemble¶
Background¶
Stacking (sometimes called stacked generalization or bagging) is an ensemble meta-algorithm that attempts to improve a model’s predictive power by harnessing multiple models (perferably different in nature) to a unified pipeline.
The Stacking method is a very general name that is sometimes used to describe different methods to crete the unfied pipeline. Here, we focus on a Stacking ensemble which uses the multiple models predict the target, while unifing them using a meta-level regressor - which learns how to annotate proper weights to the predictions of the models under it.
A simpler type of Stacking might have been to average the predictions of the different models (similar to Random Forest, but perhaps without the limitation of a single-type model).
In true Stacking the “stacker” or the meta-level regressor can also perform learning, where models which are proven to be less efficient in predicting the data are provided lower weight in the final prediction.
[1] high-level description of the stacking ensemble
Getting it Wrong¶
The major problem in creating a proper Stacking ensemble is getting it right. The wrong way to perform stacking would be to
- Train the first level models over the target.
- Get the first level models predictions over the inputs.
- Train the meta-level Stacker over the predictions of the first level models.
Why would that be the wrong way to go?
Overfitting
Our meta-level regressor would be exposed to severe overfitting from one of the first level models. For example, if one of five first level models would be highly overfitted to the target, practically “storing” the y target it is showns in train time for test time. The meta-level model, trained over the same target would see this model as excellent - predicting the target y with impressive accuracy almost everytime.
This will result in a hight weight to this model, making the entire pipeline useless in test time.
The Solution¶
The solution is never using the train abilities of the first level model - but using their abilities in test. What does it mean? it means the meta-level model would never be exposed to a y_hat generated by any first level model where the actual target sample representing this y_hat in the data was given to that model in training.
Each model will deliever its predictions in a “cross_val_predict” manner (in sklearn terms). If it’s a great model, it will demonstrate great generalization skills making its test-time predictions valuable to the meta-level regressor. If it’s a highly overfitted model - the test-time predictions it will hand down the line will be showns for their true abilities, causing it to recieve a low weight.
How do we achieve that? internal cross validation.
[1] achienving stacking ensemble using internal cross-validation