Computer Chess Club Archives




Subject: A question about statistics...

Author: Roger Brown

Date: 08:46:00 01/04/04

Hello all,

I have read numerous posts about the validity - or lack thereof actually - of
short matches between and among chess engines.  The arguments of those who say
that such matches are meaningless (Kurt Utzinger, Christopher Theron, Robert
Hyatt et al)typically indicate that well over 200 games are requires to make any
sort of statisticdal statement that engine X is better than engine Y.

I concede this point.

The arguments of the short match exponents typically centre on other
chessplaying characteristics of an engine which may also be of  interest to a
user - tactical excitement, daring, amazing moves, positional considerations,
human like play etc.

I also agree that this camp has a valid perspective.

I would like  to conduct an experiment but I need to ask a few questions first:

(1)  Is there a minimum timecontrol that is satistically relevant to games
played at classical timecontrols?  That was really one of the things I wanted to
look at but clearly it requires a pool of such games, consistent hardware, etc.

I ask this because the long timecontrol devotees have spare hardware, or at
least hardware over which they exercise an enormous amount of discretion as to
its use.  Not all of us are in that fortunate position.

Playing 200 games or more at 60 minutes + (which is still fast chess!) would
take me to a place where the light does not shine...

I am thinking that there may be a relationship - particularly as the subject is
an electronic construct - between long games and short ones.  It may not be
linear but I cannot believe that it is a coincidence that the long timecontrol
GMs are also atop the blitz ratings ladder...

(2)  What is the statistical minimum of games that I would have to play to be
able to make some sort of definitive noise?

(3)  What is the impact - or theoretical impact - of learning on such a match?
My personal bias is that if an author implements learning he should be rewarded
for it and it should be turned on at the beginning of the match.  This speaks to
positional and book learning.

(4)  I am also biased towards using the engine's particular book(s).  The
opening knowledge that a human chessplayer has is his/hers.  An engine should
have its own book with it as it goes into battle.  Can someone turn off Ms.
Polgar's opening book?  No?  Then the engine should have its book too....

(5)  The games would be played on my single processor CPU.  That would mean no
pondering *if* I understand Robert Hyatt's reasoning on the matter (which I
freely admit may not be the case at all!).

(6)  Are there any other factors?

I really would like a way to prove or disprove the position that:

(1) Games at shorter timecontrols are essentially worthless and:

(2) That matches of 1000 games are required to make statistical statements.

Please feel free to comment BUT what I would really like are some answers to the
above questions and/or pointers....


This page took 0.03 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.