Computer Chess Club Archives




Subject: Re: Proving something is better

Author: Bruce Moreland

Date: 00:28:11 12/18/02

Go up one level in this thread

On December 17, 2002 at 19:49:47, Bob Durrett wrote:

>On December 17, 2002 at 19:36:09, Vincent Diepeveen wrote:
>>On December 17, 2002 at 19:10:42, Dann Corbit wrote:
>Perhaps a useful test would be to measure how long a chess engine takes to get
>the right answer for a large set of diverse test positions.
>There would have to be some simple measure of "getting the right answer."  Maybe
>it would be sufficient to just measure the amount of time it took to obtain the
>*first* occurrence of the right answer.
>For example:  In a given test position, suppose the correct answer is c1e3.
>Then simply measure how long it took before the engine first started looking at
>A more useful measure might be to measure how long it took for the engine to
>find and keep c1e3 for a fixed amount of time, such as one minute.
>How these times are to be recorded seems to be a detail to be worked out.

This is actually a complicated issue, and we could talk about this for a very
long time.

"First occurence" is poor.

"Find and hold to end of test" is good, but if you have a program that finds all
solutions in a test suite, making the program faster can only result in fewer

Another problem with "find and hold" is that version A may find the right move
with a score of +5 in ply 9, while version B finds it with -2 in ply 8, goes to
-1.5 in ply 9, and +3 in ply 10.  How to score that is another controversial
issue.  It is possible that version A had a +3 move in ply 8, which version B
missed, or it's possible that version A might understand that white's second
move in version B's ply 8 line is lost, and finds the correct second move in ply
9, while version B needs until ply 10 to find it.


This page took 0.01 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.