Author: Christophe Theron
Date: 18:09:00 01/03/01
Go up one level in this thread
On January 03, 2001 at 12:53:35, José Carlos wrote:
>On January 03, 2001 at 12:36:51, Christophe Theron wrote:
>
>>On January 03, 2001 at 09:52:06, José Carlos wrote:
>>
>>> Lately, people have been talking here about significant results. I'm not
>>>really sure if probabilistic calculus is appropiate here, because chess games
>>>are not stocastic events.
>>> So, I suggest an experiment to mesure the probabilistic noise:
>>>
>>> -chose a random program and make it play itself.
>>> -write down the result after 10 games, 50 games, 100 games...
>>>
>>> It should tend to be an even result, and it would be possible to know how many
>>>games are needed to get a result with a certain degree of confidence.
>>> If we try this for several programs, and the results are similar, we can draw
>>>a conclusion, in comparison with pure probabilistic calculus.
>>>
>>> Does this idea make sense, or am I still sleeping? :)
>>>
>>> José C.
>>
>>
>>
>>I have done this experiment with Chess Tiger with fixed openings and reversing
>>the colors for each opening, on a large number of openings.
>
> If you use the same program and the same openings, what you're measuring is
>the effect of the program's randomness on the result. But, as normal games
>aren't played with fixed openings, there's still another thing to measure: book
>randomness effect on the results of a match.
> I guess that, with the same program and the same openings, the results would
>be pretty even from the begining. But, what I'm suggesting is, as in real games,
>let the books do their work and let the program play itself. The reason why I
>chose the same program is because, that way, I'm sure that, for a given number
>of games, the result must be very even.
> An example: if with this experiment we see that, after 1000 games, the result
>is 600-400, we'll sadly have to say that 1000 games are not enough, but if we
>see that, after 100 games, the result is 52-48, then we can safely say that 100
>games give a rather certain result.
> Of course, these examples are not absolutely correct, but show my point.
>
>>This experiment and the results I have got is the reason why I say all the time
>>that statistical significance is very important.
>
> Yes, that's my point too. That's why I suggest a way to measure what is the
>"degree of noise" depending of number of games.
As far as I can tell it is close to what you can expect with standard statistics
formulas.
Christophe
>>When you see a program beating itself 10-4, you begin to understand what I mean.
>
> That's what I wanted to test.
>
>> Christophe
>
> José C.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.