Computer Chess Club Archives


Search

Terms

Messages

Subject: How many games are needed to find out which program is stronger?

Author: Heiko Mikala

Date: 15:20:22 09/02/99

Go up one level in this thread


On September 02, 1999 at 15:52:17, Dave Gomboc wrote:


>I don't think that you addressed my point, namely:
>
>  Less games are required to conclude with a certain confidence
>  that one program is better than another when the results are
>  lopsided than when they are not.

Oh yes, I think I did exactly address your point.

Look at these two match-fragments:

    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
A   ½ 0 ½ 1 ½ 1 0 1 ½ 1 1 1 1 1 1   11.0/15
B   ½ 1 ½ 0 ½ 0 1 0 ½ 0 0 0 0 0 0    4.0/15


    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
A   1 1 ½ 1 0 0 1 0 0 1 1 1 1 1 1   10.5/15
B   0 0 ½ 0 1 1 0 1 1 0 0 0 0 0 0    4.5/15

They both look very similar, don't they? So, if I understand you correctly, you
would conclude from both matches, that engine A will most definitely be stronger
than engine B, because the results are *very* "lopsided".

The first match-fragment from above are the first 15 games of my match between
CSTal II - Hiarcs.

The second match-fragment is from the second match that I quoted in my last
post. Go back and look at the endresult of the second match, which all in all
consisted of 73 games.

My point is: you can *not* conclude from a small lopsided match result, that one
engine is stronger than the other. No matter, how "lopsided" the result is, you
will need *much* more than only 15 or so games.


Greetings,

Heiko.



This page took 0.01 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.