Author: Christophe Theron
Date: 21:40:26 01/28/00
Go up one level in this thread
On January 28, 2000 at 15:20:53, Dave Gomboc wrote:
>On January 28, 2000 at 14:44:48, Christophe Theron wrote:
>
>>On January 28, 2000 at 13:01:05, Michael Neish wrote:
>>
>>>On January 28, 2000 at 07:27:54, Enrique Irazoqui wrote:
>>>
>>>>There is a degree of uncertainty, but I don't think you need 1000 matches of 200
>>>>games each to have an idea of who is best.
>>>
>>>I have to agree with what Christophe says insofar as you need to play a
>>>certain number of games before you can determine, to a certain (known)
>>>degree of accuracy what the rating difference is between two programs.
>>>You will never know exactly of course, hence the standard deviation
>>>figures given next to the Elo ratings of human Chess players, which are
>>>sometimes overlooked. I haven't read Elo's book, but from what I know
>>>of the Elo system he must have taken all this probability stuff into
>>>account when he formulated it, so meaningless it is not. In fact, it is
>>>the core of the entire system.
>>>
>>>If the rating difference between two programs is quite small, say less than
>>>35 points, then I'm afraid you will definitely need a lot of games to sort it
>>>out from the results alone. A 20-game match solves nothing. Christophe,
>>>if you're reading this, could you tell us what is the minimum Elo difference
>>>that a 20-game match can estimate to a good degree of confidence?
>>
>>
>>With a 20 games match, you can determine if prog A is 77 elo points above prog
>>B, with a 80% confidence.
>>
>>If the programs are closer in ELO, the 20 games match is not enough.
>>
>>I have answered to Enrique and given a complete table in my answer, you might
>>find it very interesting.
>>
>>The key point is that when the elo difference gets smaller, the number of games
>>to play increases tremendously.
>
>...and 80% confidence is terrible. 95% is usually used. There'd have to be an
>enormous strength difference for a 20-game match to be reasonably conclusive.
>
>Dave
I wouldn't disagree that 80% confidence is terrible. But I clearly stated that
my table was for this interval of confidence, isn't it?
I would be happy if we got tournament results with a 80% confidence. But it's
not even the case.
You can use RNDMATCH.BAS to compute numbers for 95% confidence if you want.
Or you can compute them with statistic formulas, which would be better. I hope
you'd get almost the same results.
Christophe
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.