Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Small number statistics and small differences

Author: Robert Henry Durrett

Date: 06:05:21 08/15/98

Go up one level in this thread


On August 14, 1998 at 11:36:56, Dan Homan wrote:

<snip>
>>>>>>Someone who has more statistics than I may be willing to comment on this.
OK
<snip>
>Of course, but my point is that if you think you are learning anything
>useful from 4-0 result in a 4 game match between top programs you are
>wrong.  I initially thought this was a great idea until I thought about
>it - the more I think about it the worse it gets.
<snip>
(1)Every time you run an experiment you get some information.  This is true even
if you have only one "toss of the coin." [It just may not be what you were
hoping for.  You may have hoped to get nearly absolute confidence in a result
you were fishing for.]
(2)For example, for a given experiment [ex:  so many games ("trials"), wins,
losses & draws] under specified conditions [opening books used, etc.] then the
results could be given as a plot of probabilities [and confidence levels] for
each possible playing strength difference [expressed as rating differences.]
For "Computer A is 100 points stronger than computer B" you get one probability.
 For "Computer A is 300 points weaker than conputer B" you get another
probability.  You can continue along these lines for all possible rating
differences. You could plot these on a piece of graph paper.
(3)Generally, the more trials, the more confidence.  But this is probability,
not absolute conclusions.  If the weather man says 80% chance of rain, you still
don't know for sure, but you will carry your umbrella anyway.
(4)Small numbers of trials give very little information about extreme cases!
For example, if you wish to know if computer A is 1 or 2 rating points stronger
or weaker than computer B, you will have to have many trials before you can have
any confidence in your conclusion.  Similiarly, if you wish to know "Is Computer
A is 900 points stronger than Computer B or is it 1000 points stronger" you will
have to have many trials to resolve that issue with any confidence.



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.