Author: Vasik Rajlich
Date: 01:52:11 08/31/05
Go up one level in this thread
On August 30, 2005 at 12:27:52, Peter Berger wrote: >On August 30, 2005 at 12:21:20, Maurizio De Leo wrote: > >> >>>Under valid and controlled conditions it still seems logical to me to stop a >>>test after a 5-0 result and conclude that the winning program is probably the >>>stronger one. >> >>>>I don't put much credence in any result of less than 30 games. >>>>After 30 games, then you get a lot more plausibility. >> >>>You didn't give any reason for this, so I don't understand. A 6-0 says more >>>about engine strength than the above match result with over 100000 games. >> >>Dann is right, I think. >>The confidence interval calculation assumes that the score of a game is a >>statistic variable with a mean value between 1 and -1 (function of the Elo >>difference between the programs) and a standard deviation. Then if the >>experiments are independent, the sum of the points will approximate the product >>(mean*number of games) with a smaller standard deviation the more the games are. >>With enough games the "confidence" will get to 95% when the performance >>difference between the two programs is more than 3 standard deviations. >>However this assumes a normal distribution. The assumption can be made for any >>repeated statistical variable as long as the experiments are independent and >>"enough". This "enough" is indeed expressed in most statistics books as 30. >> >>Maurizio > >Please have a look at "WhoisBest.zip" at Rémi Coulom's Home Page: >http://remi.coulom.free.fr/. It includes a little paper Whoisbest.pdf on >"Statistical Significance of a Match" , with a very straightforward mathematical >proof that for example the number of draws is irrelevant to conclude who is >better in a chessmatch . > >Peter It's not that simple, due to the nature of chess. In chess, a match result of 2-0 with 0 draws is less significant than a match result of 2-0 with 8 draws. WhoIsBest makes the assumption that draws are independent events - that is, that wins, losses and draws each come with some independent probability. In fact, in a +2 -0 =8 result, the chance is that the side with the +2 was "stronger" in the draws - ie. closer to winning. Chess has this phenomenon where the stronger side tries to break through the draw barrier, and sometimes cannot. Of course to model this mathematically would be a huge mess. Vas
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.