Author: martin fierz
Date: 13:44:06 09/28/04
Go up one level in this thread
On September 28, 2004 at 14:04:06, Robert Hyatt wrote: [snip] >If you can run the same test set, same program, and get different results, it >points out a _serious_ flaw in your testing methodology. Search to fixed depth >and that will go away. Or search for 1 second but _always_ finish the last >iteration. The variance will disappear as well. i don't think the second method is any good. search to fixed depth => it will go away, i agree. but as soon as you say "search for 1 second and then X" it won't work any more. if you have a timing problem when aborting the search at exactly 1 second, you will have the same timing problem if you add the "and then X" part to it. in your example, once he will terminate a search after 0.999 seconds, the other time it will take 1.000 seconds and need another iteration. i think something else is much worse: even if stuart were to use fixed depth, his testing (WAC)is still useless. if you take a long time to tune a program to excel at a test set, it will be mostly coincidential that it performs well there. add a really beneficial change, and chances are good that the test set solution rate will drop, since it is not perfectly tuned to coincidentally produce many solutions.... test suites are a great way of quickly checking whether you have broken something in your code. they are also a good way of measuring tactical progress if you *don't* tune to the test set, ever. for anything else, they are useless IMO. however, these two items are quite useful. cheers martin
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.