Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: How many games are needed to find out which program is stronger?

Author: Bruce Moreland

Date: 18:40:37 09/04/99

Go up one level in this thread


On September 04, 1999 at 21:05:06, James T. Walker wrote:

>Well, don't put words in my mouth.  If I had a score of 502-498 I could say that
>they are very even with a high degree of confidence.  I would not be stupid
>enough to say that "A is stronger than B".  However if I had a score of 5.0 to
>5.0 I would not be so confident that they were even.  That's what I mean by
>having a higher probability that your conclusion would be correct.  By the same
>reasoning if I had 600-400 I would be more confident that A is better than B if
>the score were only 6-4.  So I stick to my statement.  More data gives you a
>higher degree of certainty.  (Just don't mis-interpret the data)

It depends upon what you conclude.  If the ratio of wins to losses is the same,
more data means you can make a more certain conclusion, unless the score is
exactly 50%.  But if the match is closer, it may be possible that you can say "A
is better than B" with less confidence.

>>If you play 100 games, and A beats B by a score of 100-0, the statement "A is
>>stronger than B", is a very weak statement, it is a conservative conclusion.
>>The odds that A really is stronger than B are almost 100% in this case, whereas
>>in the previous case the odds that A is stronger than B are only slightly more
>>than 50%.
>
>There would be very little argument in this case but again you are interpreting
>the data incorrectly to make your point.  But "A is stronger than B" is
>perfectly valid and nothing weak about it.  If you said "Maybe A is better than
>B" that's a weak statement but I wouldn't say that given the above data(100-0).
>Even in the face of a small probability that the next 100 might be 0-100.

What I mean by "weak" is that it is an understatement, and sorry if I am using
the wrong terminology.  I am saying that if you weaken what you say, meaning
that if you make what you say more conservative, the probability that it is
correct increases.

bruce




This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.