Author: Albert Silver
Date: 05:56:52 02/02/06
Go up one level in this thread
On February 01, 2006 at 17:49:03, Vasik Rajlich wrote:
>On January 30, 2006 at 18:14:21, Albert Silver wrote:
>
>>>Let me put it like this: that is the chance at this point (ie. just based on the
>>>games you list above, without any further testing) that your hypothesis is
>>>correct?
>>>
>>>If you analyze it "statistically", you might get some figure like (let's say)
>>>30%. Just a total wild guess, eyeballing your numbers.
>>>
>>>In reality, though, it's probably more like 3%.
>>>
>>>The reason is that before your experiment started, there were let's say 20
>>>candidate hypotheses, that you didn't even bother to list. Maybe ultrasolid is
>>>worse against Fritz, maybe it's better in closed positions, etc. One of these
>>>hypetheses is likely to get lucky, and this hypothesis will then of course have
>>>very nice data to support it.
>>>
>>>Anyway, there is nothing wrong with this procedure, as long as you eventually
>>>test your hypothesis "straight up".
>>>
>>>Vas
>>
>>By "straight up", do you mean it is tested alone without any other parameters
>>influencing? If so, wouldn't that go against the theory that each parameter is
>>independent of the other and should bring its fruits?
>>
>
>What I mean by straight up is that the testing should go in the following
>sequence:
>
>1) Play a bunch of games, with various settings, without any special
>expectations
>2) Identify some trend - let's call it a "candidate hypothesis"
>3) Test the candidate hypothesis with many many more games
>
>Maybe, to be really fair, the games from step 1 should even be thrown away, and
>only the games from step 3 should be used. Not sure about this.
>
>Consider the following scenario.
>
>There is some person (Joe) who just won the lottery. You want to see if Joe is a
>specialist at winning lotteries. So, Joe buys many more lottery tickets, and we
>see how he performs. The question is: is it fair to include that first win in
>his statistics, or not? Probably not.
>
>Vas
Unless the testing conditions are identical (time control and openings), in
which case there is no need to repeat the result. Otherwise, your description
above is exactly what I had in mind, so we're seeing eye to eye here.
Albert
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.