Book Blurbs » Classification and Regression Trees by Breiman et al http://books.hammerpig.com Quotes to Remember From Some Great Books About Science, People, Technology, and Ideas Tue, 12 May 2009 16:53:08 +0000 http://wordpress.org/?v=2.9.1 en hourly 1 Impurity in Tree Classifiers http://books.hammerpig.com/impurity-in-tree-classifiers.html http://books.hammerpig.com/impurity-in-tree-classifiers.html#comments Wed, 11 Feb 2009 15:37:04 +0000 admin http://books.hammerpig.com/?p=38 “The first problem in tree construction is how to use L to determine the binary splits of X into smaller and smaller pieces. The fundamental idea is to select each split of a subset so that the data in each of the descendant subsets are ‘purer’ than the data in the parent subset…the node impurity is largest when all classes are equally mixed together in it, and smallest when the node contains only one class.” (pp. 23-24)

]]> http://books.hammerpig.com/impurity-in-tree-classifiers.html/feed 0 The Gini Index http://books.hammerpig.com/the-gini-index.html http://books.hammerpig.com/the-gini-index.html#comments Wed, 11 Feb 2009 15:29:27 +0000 admin http://books.hammerpig.com/?p=34 “The concept of a criterion depending on a node impurity measure has already been introduced. Given a node t with estimated class probabilities p(j|t), j=1, …, J, a measure of node impurity given t:

i(t) = psi(p(1|t), …, p(J|t))

is defined and a search made for the split that reduces node, or equivalently tree, impurity. As remarked earlier, the original function selected was:

psi(p1, …, pJ) = -Sum(j)(pj * log(pj)).

“In later work the Gini diversity index was adopted. This has the form:

Sum (j!=i) (p(i|t)p(j|t)).

“The Gini index has an interesting interpretation. Instead of using the plurality rule to classify objects in a node t, use the rule that assigns an object selected at random from the node to class i with probability p(i|t). The estimated probability that the item is actually in class j is p(j|t). Therefore, the estimated probability of misclassification under this rule is the Gini index:

Sum (j!=i) (p(i|t)p(j|t)).

“Another interpretation is in terms of variances (see Light and Margolin, 1971). In a node t, assign all class j objects the value 1, and all other objects the value 0. Then the sample variance of these values is p(j|t)(1-p(j|t)). If this is repeated for all J classes and the variances summed, the results is:

Sum(j) (p(j|t)(1-p(j|t)) = 1 – Sum(j) (p^2(j|t))).

…

The Gini index is simple and quickly computed. It can also incorporate symmetric variable missclassification costs in a nature way.” (pp. 103-104)

]]> http://books.hammerpig.com/the-gini-index.html/feed 0