Computer Chess Club Archives




Subject: Re: A question about statistics...

Author: Mark Young

Date: 10:14:22 01/04/04

Go up one level in this thread

On January 04, 2004 at 12:48:52, Roger Brown wrote:

>On January 04, 2004 at 12:29:15, Mark Young wrote:
>>On January 04, 2004 at 11:46:00, Roger Brown wrote:
>>If you concede this point you don't understand. There is no magic number like
>>200 or 2000. The score must be considered. Here is an example:
>>A score of 17 - 3 in a 20 game match has a certainty of over 99% that the winner
>>of the match is stronger then the loser.
>>A 100 game match ending 55 - 45 only has a 81% chance that the winner of the
>>match is the stronger program.
>>A 200 game match ending 106 - 94 only has a 78 % chance that the winner is
>>stronger then the loser.
>I hear you.  However the point that the 200 games advocates make is that, with
>respect to the 17-3 score over a 20 game match, the result is open to question
>because another batch of twenty games could produce an entirely different
>Kurt Utzinger published breakdown figures in match between Gandalf and a beta
>engine he is/was testing and it ended up statistically even.  Had he stopped it
>earlier, he would have erroneously concluded that X was stronger than Y.
>I have observed that myself as well...

You can never conclude anything for sure, not ever with 200 games. You need to
find a level of confidence that you can live with.

My point being there is no need to run hundereds of games once you reach that
level. Time is a factor to consider. I for one will not waste weeks or month
playing 200 games at 40 moves in 2 hours, once the results have reach my
confidence level.


This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.