Author: J. Wesley Cleveland
Date: 17:46:01 09/29/04
Go up one level in this thread
On September 28, 2004 at 16:44:06, martin fierz wrote: >On September 28, 2004 at 14:04:06, Robert Hyatt wrote: >[snip] > > >>If you can run the same test set, same program, and get different results, it >>points out a _serious_ flaw in your testing methodology. Search to fixed depth >>and that will go away. Or search for 1 second but _always_ finish the last >>iteration. The variance will disappear as well. > >i don't think the second method is any good. search to fixed depth => it will go >away, i agree. but as soon as you say "search for 1 second and then X" it won't >work any more. if you have a timing problem when aborting the search at exactly >1 second, you will have the same timing problem if you add the "and then X" part >to it. in your example, once he will terminate a search after 0.999 seconds, the >other time it will take 1.000 seconds and need another iteration. > >i think something else is much worse: even if stuart were to use fixed depth, >his testing (WAC)is still useless. if you take a long time to tune a program to >excel at a test set, it will be mostly coincidential that it performs well >there. add a really beneficial change, and chances are good that the test set >solution rate will drop, since it is not perfectly tuned to coincidentally >produce many solutions.... > >test suites are a great way of quickly checking whether you have broken >something in your code. they are also a good way of measuring tactical progress >if you *don't* tune to the test set, ever. for anything else, they are useless >IMO. however, these two items are quite useful. > >cheers > martin In general I agree about test sets, but here we are not tuning evaluations or extensions. We expect exactly the same scores and PVs. We are looking for the best way to order root moves to reduce the number of nodes searched. I see no reason why this should be different for test sets and games, though it might be worth testing quiet positions where the best move changes to see.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.