Author: Mike Byrne
Date: 10:07:30 01/04/04
Go up one level in this thread
On January 04, 2004 at 13:03:04, Roger Brown wrote: >>When you play 100 games, the confidence level will be near 95% +/- 60 points. >>So you end up with a difference that is greater than 120 points you say with 95% >>confidence that the higher rated program is better than the lower rated program. >> So there would be a 1 out of 20 chance that your test results are not correct. >> If they are rated less than 120 points apart, statically, you can make a claim >>either way, you need more games. The closer the rating is , the more games you >>need to achive a 95% confidence level, > > > >So it is a hundred games that you suggest. Two hundred should be even better >then. > Statistically, the more games the better - yes - your confidence level increases with more games at a give +/- range. Or you can decerease the +/- range and keep your confidence level the same - either way - more games make it better. > > >> >>Statistically, the 100 games does have NOT to be against the same engine,, You >>can design a tournament with 11 engines, have them play round robin with rounds >>(10 cycles) - and you confidence level for the ratings relelative to each is >>just as valid as single 100 game match. Alternativley, you can play 100 rounds >>with 101 engines, they each play one game with each other - statisically it is >>the same. The key is simply 100 games. Think about human ratings, you do not >>play 100 games against the same player, you might play 100 games against 90 >>players - statistically it does not matter. > > > >Agreed but, *ahem* I do not have a sufficient number of engines to play 100 >different games. Your point is dramatically made though. > >I want to use engines near the top - for reasons of development, stability of >use, substantial debugging and testing. In short engines like Crafty, Yace and >Little Goliath which are featured in several tournaments and can be used with a >robust, automatic and computer friendly (memory, cpu usage etc.) gui like >Winboard. > > >I want no issues with draw claims and the like.... > > >:-) > > >Thanks for the feedback - and for the modified Crafties! > > >Later.
This page took 0.04 seconds to execute
Last modified: Thu, 07 Jul 11 08:48:38 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.