Author: Bruce Moreland
Date: 00:28:11 12/18/02
Go up one level in this thread
On December 17, 2002 at 19:49:47, Bob Durrett wrote: >On December 17, 2002 at 19:36:09, Vincent Diepeveen wrote: > >>On December 17, 2002 at 19:10:42, Dann Corbit wrote: >> ><snip> > >Perhaps a useful test would be to measure how long a chess engine takes to get >the right answer for a large set of diverse test positions. > >There would have to be some simple measure of "getting the right answer." Maybe >it would be sufficient to just measure the amount of time it took to obtain the >*first* occurrence of the right answer. > >For example: In a given test position, suppose the correct answer is c1e3. >Then simply measure how long it took before the engine first started looking at >c1e3. > >A more useful measure might be to measure how long it took for the engine to >find and keep c1e3 for a fixed amount of time, such as one minute. > >How these times are to be recorded seems to be a detail to be worked out. This is actually a complicated issue, and we could talk about this for a very long time. "First occurence" is poor. "Find and hold to end of test" is good, but if you have a program that finds all solutions in a test suite, making the program faster can only result in fewer solutions. Another problem with "find and hold" is that version A may find the right move with a score of +5 in ply 9, while version B finds it with -2 in ply 8, goes to -1.5 in ply 9, and +3 in ply 10. How to score that is another controversial issue. It is possible that version A had a +3 move in ply 8, which version B missed, or it's possible that version A might understand that white's second move in version B's ply 8 line is lost, and finds the correct second move in ply 9, while version B needs until ply 10 to find it. bruce
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.