Author: Uri Blass
Date: 22:43:17 01/31/01
Go up one level in this thread
On January 31, 2001 at 19:29:55, Bruce Moreland wrote:
>On January 31, 2001 at 15:37:21, Dann Corbit wrote:
>
>>On January 31, 2001 at 14:17:48, Bruce Moreland wrote:
>>[snip]
>>>If you start a match and get 10-0 right away, it proves that p is bigger, by any
>>>reasonable standard of proof.
>>
>>And yet the SSDF has had matches start out that way which went to the other
>>opponent in the end (or something fairly close to that -- I forget the exact
>>figures for an O-fer reversal).
>>
>>With chess, the odds of 0/10 for evenly matched chess engines is harder to
>>figure, but with a coin toss it is easy:
>>
>>1/(2^10){all heads} + 1/(2^10){all tails} = 1/(2^9) = .2%
>>
>>Hence, if you had one thousand people flip ten pennies, (on average) two of them
>>would get either all heads or all tails. The question is -- are you one of
>>those people when you run an experiment?
>>
>>Improbable events do happen. That's why we buy fire insurance.
>>;-)
>
>You will have improbable cases occur. That is why they are called improbable
>and not impossible.
>
>I don't know why people have a hard time dealing with this issue. Every match
>result has associated with it a probability that the result could be achieved by
>two equal programs, purely by chance.
>
>If you get a result, and declare that the winner of the match is the better of
>the two programs, there is a chance that you will be wrong.
>
>If you play more games it is not guaranteed that this chance is reduced.
>
>What I can't understand is why people look at a 10-0 result and say, "That is
>due to chance!", and look at a 60-40 result and say, "One of the programs is
>better!", when the probability that the second result is due to chance is
>greater than the probability that the first result is due to chance.
I agree that the probability to get 10-0 is smaller than the probability to get
60-40 result with equal programs.
The problem is that it does not mean that in cases when you see 10-0 result you
can be more sure that the winner is better than in cases that you see 60-40
result.
The probability that the winner is better after you see 10-0 result is a
different probability than the probability to see 10-0 result when the programs
are equal.
In order to know the probability that the winner is better after you see 10-0
result tou need to have some assumptions about the probability of the winner to
win.
For example you may assume before testing that the probability of one side to
win has a uniform distribution and can get any value between 0 and 1.
In this case the probability of the better side to win 10-0 is calculated by an
integral (the erea behind the function p^10 when 1/2<p<1 multiply by 2) and it
is 2/11-1/(1024*11).
The probability of the weaker side to win 10-0 is 1/(1024*11)
If I have no mistake in my calculations it means that the pobability of the
winner to be the better player is 2047/2048.
I did not clculate the same for the 60-40 result but I believe that in this case
the probability of the better side to win is smaller so 10-0 is more
significant.
The problem when there is a small change is that you know before the test that
the distribution is not like this and it is more logical to assume that
probability of one side to win has a uniform distribution with a smaller range
of values and I suspect that in these cases 60-40 will be more significant.
I think that waiting to a constant difference is a good idea because practically
if we need a lot of games to decide which is stronger the difference is probably
smaller so it is less important for us to be right about the stronger after many
games.
Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.