Author: Robert Hyatt
Date: 13:16:01 03/06/01
Go up one level in this thread
On March 06, 2001 at 14:58:24, Dann Corbit wrote: >On March 05, 2001 at 22:51:19, Robert Hyatt wrote: > >>On March 05, 2001 at 20:59:05, Dann Corbit wrote: >> >>>On March 05, 2001 at 20:41:17, Robert Hyatt wrote: >[snip] >>>If you are trying to produce a reproducable experiment, then you don't change >>>the parameters as you go. If you just want a fun contest, then do whatever you >>>want. >> >>Tournaments are definitely _not_ reproducible. In any shape, form or >>fashion... either human events, nor computer events where the authors >>are present. > >By the same token, a sequence of 1000 coin flips won't be reproducable either. >Any measurement with a degree of randomness will suffer from this problem (which >is *truthfully* -- ALL of them). At any rate, if you are trying to *win* a >contest, then you will try anything at your disposal. Certainly you can get >some gains by being unpredictable (e.g. changing the openings or whatever). But >if the experiment is planned to measure something and produce a number, then you >should eliminate as many variables as possible. With software this is simply impossible to eliminate. IE if you use crafty version X to play in a tournament, even after version X+1 is out, then you are already introducing a random variable to the tournament. Because if you hold another one immediately after this one and use the then-current versions of everything you will get different results. You will probably get different results if you use _exactly_ the same engines, so worrying about a few bug fixes is really about like trying to optimize a piece of code that takes less than .0000001% of the total search time. Any changes won't make a difference there.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.