Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: A question about statistics...

Author: Ricardo Gibert

Date: 09:40:00 01/04/04

Go up one level in this thread


On January 04, 2004 at 12:29:15, Mark Young wrote:

>On January 04, 2004 at 11:46:00, Roger Brown wrote:
>
>>Hello all,
>>
>>I have read numerous posts about the validity - or lack thereof actually - of
>>short matches between and among chess engines.  The arguments of those who say
>>that such matches are meaningless (Kurt Utzinger, Christopher Theron, Robert
>>Hyatt et al)typically indicate that well over 200 games are requires to make any
>>sort of statisticdal statement that engine X is better than engine Y.
>>
>>I concede this point.
>
>If you concede this point you don't understand. There is no magic number like
>200 or 2000. The score must be considered. Here is an example:
>
>A score of 17 - 3 in a 20 game match has a certainty of over 99% that the winner
>of the match is stronger then the loser.
>
>A 100 game match ending 55 - 45 only has a 81% chance that the winner of the
>match is the stronger program.
>
>A 200 game match ending 106 - 94 only has a 78 % chance that the winner is
>stronger then the loser.


Nothing you have said is really correct because you have ignored the significant
effect of draws in a match.


>
>
>>
>>The arguments of the short match exponents typically centre on other
>>chessplaying characteristics of an engine which may also be of  interest to a
>>user - tactical excitement, daring, amazing moves, positional considerations,
>>human like play etc.
>>
>>I also agree that this camp has a valid perspective.
>>
>>I would like  to conduct an experiment but I need to ask a few questions first:
>>
>>(1)  Is there a minimum timecontrol that is satistically relevant to games
>>played at classical timecontrols?  That was really one of the things I wanted to
>>look at but clearly it requires a pool of such games, consistent hardware, etc.
>>
>>I ask this because the long timecontrol devotees have spare hardware, or at
>>least hardware over which they exercise an enormous amount of discretion as to
>>its use.  Not all of us are in that fortunate position.
>>
>>Playing 200 games or more at 60 minutes + (which is still fast chess!) would
>>take me to a place where the light does not shine...
>>
>>I am thinking that there may be a relationship - particularly as the subject is
>>an electronic construct - between long games and short ones.  It may not be
>>linear but I cannot believe that it is a coincidence that the long timecontrol
>>GMs are also atop the blitz ratings ladder...
>>
>>
>>(2)  What is the statistical minimum of games that I would have to play to be
>>able to make some sort of definitive noise?
>>
>>
>>(3)  What is the impact - or theoretical impact - of learning on such a match?
>>My personal bias is that if an author implements learning he should be rewarded
>>for it and it should be turned on at the beginning of the match.  This speaks to
>>positional and book learning.
>>
>>
>>(4)  I am also biased towards using the engine's particular book(s).  The
>>opening knowledge that a human chessplayer has is his/hers.  An engine should
>>have its own book with it as it goes into battle.  Can someone turn off Ms.
>>Polgar's opening book?  No?  Then the engine should have its book too....
>>
>>
>>(5)  The games would be played on my single processor CPU.  That would mean no
>>pondering *if* I understand Robert Hyatt's reasoning on the matter (which I
>>freely admit may not be the case at all!).
>>
>>
>>(6)  Are there any other factors?
>>
>>
>>
>>I really would like a way to prove or disprove the position that:
>>
>>(1) Games at shorter timecontrols are essentially worthless and:
>>
>>(2) That matches of 1000 games are required to make statistical statements.
>>
>>
>>Please feel free to comment BUT what I would really like are some answers to the
>>above questions and/or pointers....
>>
>>
>>
>>Later.



This page took 0.06 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.