Tight bounds for the probability of overfitting

A combinatorial approach is developed that leads to tight bounds for the probability of overfitting in a number of special cases. The Vapnik Chervonenkis classical bound is easy to restate under the weak probabillity assumptions where △ is the diversity coefficient of A, which is equal to the number of different error vectors generated by all possible algorithms a from A. An experimental analysis of major causes of overestimated bound shows that the probability of over fitting depends substantially not only on the number of different error vectors but also on the degree of their difference. The set A may contain a large number of pairs of similar algorithms. Specifically, most classification algorithms used in practice have a separating surface that is continuous with respect to the parameters.