Bagging follows three stages: (1) create $$B$$ bootstrap training sets by sampling observations from the original training data with replacement; (2) independently fit the same base learning method to each bootstrap set; and (3) combine their predictions. Regression predictions are typically averaged, while classification predictions are typically combined by majority vote. This aggregation generally produces a more stable prediction than a single fitted model.

University of Michigan - Ann Arbor

Google

Bootstrap aggregation, or bagging, is a way to help construct more powerful prediction models by reducing high variance of a statistical learning method. It is often used to improve the performance of decision trees. In order to reduce the variance, $$B$$ different bootstrapped training data sets are produced (typically taken from a single training data set). It then uses the $$b$$th bootstrapped training data set to train the method to produce $$\hat{f}^{*b}(x)$$. Afterwards, all the predictions are then averaged in order to get $$\hat{f}_{bag}(x) = \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x)$$.

Bagging

Bagging can be applied to decision trees, particularly with regression. It follows the same overall process: B regression trees are produced using B bootstrapped training sets. Afterwards, the predictions of all B regression trees are averaged in order to obtain a model that has reduced the variance of the B individual decision trees.

Bagging for Regression Trees

Bagging (bootstrap aggregation) can be applied to classification trees to improve prediction accuracy. While various approaches exist, the simplest involves recording the class predicted by each of the $$B$$ individual classification trees for a given test observation. The final prediction is determined by a majority vote, assigning the observation to the most commonly predicted class among the $$B$$ predictions.

Bagging for Classification Trees

Out-of-Bag (OOB) error estimation is a technique used to evaluate the test error of a bagged model without requiring cross-validation. An observation is considered 'out-of-bag' if it was not used to fit a given bagged tree. On average, each observation is OOB for approximately $$B/3$$ of the $$B$$ total trees. To predict the response for the $$i$$th observation, the model uses only the subset of trees where that observation was OOB. For regression trees, these specific OOB predictions are averaged to yield a single prediction for the $$i$$th observation, whereas for classification trees, a majority vote is taken. Finally, these predictions are compared against the true responses to calculate the overall OOB error, such as the Mean Squared Error (MSE) for regression or the classification error rate.

Out-of-Bag Error Estimation

*Random forests* is quite similar to *bagging* with *decision trees* except that there is an improvement on spliting trees. They both use bootstrapped subset of training oberservations for a tree and repeated that process several times for the average/most common result. 

The difference lies in that, random forest only use randomly selected subsets of all predictors for spliting trees. This improvement *decorrelates* the trees and solves the problem of bagging that,  if there is a very strong predictor, most bagged trees might look similar and averaging could not reduce much variance for the highly correlated quantities. However, this can be addressed by decorrelating the trees thus reducing the variance and making the trees more reliable.

Random Forests vs. Bagging with Decision Trees

Boosting is heavily dependent on the trees formed before it, whereas in bagging, each tree is formed independently and varies instead based on the sample of predictors chosen. 

Boosting vs Bagging

Below is a video that I found helpful about understanding bagging and decision trees. The coursera course the video is a part of may be helpful as well.

[SAS]. Bagging [Video File]. Retrieved from [www.coursera.org/lecture/machine-learning-sas/bagging-Rj2Fq](https://www.coursera.org/lecture/machine-learning-sas/bagging-Rj2Fq) 

Coursera: Bagging using Decision Trees

While bagging improves the prediction accuracy of decision trees, it reduces model interpretability because the ensemble can no longer be represented as a single tree. Variable importance measures address this by calculating the significance of predictors across all $$B$$ bagged trees. For regression trees, a variable's importance is determined by the total amount the Residual Sum of Squares (RSS) is decreased due to splits over that predictor, averaged over all $$B$$ trees. For classification trees, importance is measured by the total decrease in the Gini index caused by splits over the specific predictor, averaged across all $$B$$ trees. A larger average decrease indicates a more important predictor.

Learn Before

Related