In the context of decision trees, the classification error rate is the fraction of training observations in a given region that do not belong to the most common class. It is mathematically defined as: $$ E = 1 - \max_{k} \left(\hat{p}_{m k}\right) $$ where $$\hat{p}_{m k}$$ represents the proportion of training observations in the $$m$$th region that belong to the $$k$$th class. While classification error rate is a natural metric, it is not sufficiently sensitive for tree-growing (splitting), making other measures like Gini index or entropy preferable in practice.

University of Michigan - Ann Arbor

Claude

Google

Unlike regression trees, RSS cannot be used as a criterion for making binary splits in classification trees since the most commonly classes, rather than mean values, are used as the predictions. Three criteria that can be used for making binary splits in classification trees are

- classification error rate
- Gini index
- entropy

Criteria for making binary splits in a classification tree

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Dive into Deep Learning

Quite similar to Gini index numerically, entropy can also reflects the node purity. A samll value implies that one class dominates the region.
$$
D=-\sum_{k=1}^{K} \hat{p}_{m k} \log \hat{p}_{m k}
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class. Entropy also measures variance in the label per features.


Entropy in information Theory

Gini index measures the node purity - total variance across all the K classes. A small value of Gini index implies that one class dominates the region.
$$
G=\sum_{k=1}^{K} \hat{p}_{m k}\left(1-\hat{p}_{m k}\right)
$$
where $p_{mk}$ denotes the proportion of training observations in the $m$th region that are from the $k$th class.
The index measures total variance across the $$K$$ classes. The index is minimized when $$p_{mk}$$ is close to zero or 1. 


Learn Before

Related