# Computer Chess Club Archives

## Messages

### Subject: Re: Proving something is better

Author: Bruce Moreland

Date: 00:28:11 12/18/02

Go up one level in this thread

```On December 17, 2002 at 19:49:47, Bob Durrett wrote:

>On December 17, 2002 at 19:36:09, Vincent Diepeveen wrote:
>
>>On December 17, 2002 at 19:10:42, Dann Corbit wrote:
>>
><snip>
>
>Perhaps a useful test would be to measure how long a chess engine takes to get
>the right answer for a large set of diverse test positions.
>
>There would have to be some simple measure of "getting the right answer."  Maybe
>it would be sufficient to just measure the amount of time it took to obtain the
>*first* occurrence of the right answer.
>
>For example:  In a given test position, suppose the correct answer is c1e3.
>Then simply measure how long it took before the engine first started looking at
>c1e3.
>
>A more useful measure might be to measure how long it took for the engine to
>find and keep c1e3 for a fixed amount of time, such as one minute.
>
>How these times are to be recorded seems to be a detail to be worked out.

long time.

"First occurence" is poor.

"Find and hold to end of test" is good, but if you have a program that finds all
solutions in a test suite, making the program faster can only result in fewer
solutions.

Another problem with "find and hold" is that version A may find the right move
with a score of +5 in ply 9, while version B finds it with -2 in ply 8, goes to
-1.5 in ply 9, and +3 in ply 10.  How to score that is another controversial
issue.  It is possible that version A had a +3 move in ply 8, which version B
missed, or it's possible that version A might understand that white's second
move in version B's ply 8 line is lost, and finds the correct second move in ply
9, while version B needs until ply 10 to find it.

bruce

```