Author: Mark Young
Date: 10:14:22 01/04/04
Go up one level in this thread
On January 04, 2004 at 12:48:52, Roger Brown wrote: >On January 04, 2004 at 12:29:15, Mark Young wrote: > >>On January 04, 2004 at 11:46:00, Roger Brown wrote: >>If you concede this point you don't understand. There is no magic number like >>200 or 2000. The score must be considered. Here is an example: >> >>A score of 17 - 3 in a 20 game match has a certainty of over 99% that the winner >>of the match is stronger then the loser. >> >>A 100 game match ending 55 - 45 only has a 81% chance that the winner of the >>match is the stronger program. >> >>A 200 game match ending 106 - 94 only has a 78 % chance that the winner is >>stronger then the loser. > > > >I hear you. However the point that the 200 games advocates make is that, with >respect to the 17-3 score over a 20 game match, the result is open to question >because another batch of twenty games could produce an entirely different >profile. > >Kurt Utzinger published breakdown figures in match between Gandalf and a beta >engine he is/was testing and it ended up statistically even. Had he stopped it >earlier, he would have erroneously concluded that X was stronger than Y. > >I have observed that myself as well... You can never conclude anything for sure, not ever with 200 games. You need to find a level of confidence that you can live with. My point being there is no need to run hundereds of games once you reach that level. Time is a factor to consider. I for one will not waste weeks or month playing 200 games at 40 moves in 2 hours, once the results have reach my confidence level. > > >Later.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.