Computer Chess Club Archives




Subject: Re: Proving something is better

Author: Robert Hyatt

Date: 13:20:16 12/18/02

Go up one level in this thread

On December 18, 2002 at 15:41:11, Bruce Moreland wrote:

>On December 18, 2002 at 14:47:00, Robert Hyatt wrote:
>>On December 18, 2002 at 13:47:41, Gian-Carlo Pascutto wrote:
>>>On December 18, 2002 at 13:10:09, Robert Hyatt wrote:
>>>>Time by itself is also misleading.
>>>>It would not be hard to search the PV quickly and then bog down on refuting
>>>>the rest of the moves, which would allow time-only to produce a false
>>>How is this going to give you faster times exactly?
>>If you search to time of solution, you get one time.  If you search to the
>>end of that iteration, you get a much _longer_ time.  If the algorithm breaks
>>the search as I explained, either time is misleading by itself...
>The use of time can be misleading, but that is no reason to ignore it and give
>every algorithm as much time as it wants when trying to ascertain performance.
>The eventual goal is to find more stuff faster.  The way to tell if something is
>faster than something else is to have a race.
>Of course, it is possible to argue that some distances for the race are stupid.
>It makes no sense to have a 1 centimeter sprint.  This does not mean that the
>whole idea of a race is wrong.

That's not what I was talking about.  There are two ways you can compare
two chess algorithms, logically.

1.  time until it finds the solution move with the right score.

2.  time until it completes a particular search depth.

If you are playing with search extensions (or de-extensions in the case of
null-move) 2 has an obvious problem.  Because search depths don't mean much
when you are trying to find a move.

If you are playing with search extensions or whatever, it is possible to
greatly change the shape of the search tree, so that time to solution looks
good, but time to depth looks bad, or vice-versa.

I'm personally not convinced there is _any_ way to _really_ compare two
algorithms except to play them in a long match.  Tactically stronger !=
stronger overall, in all cases.  Tactically weaker != weaker overall, in
all cases.

I think that for times, _both_ approaches are needed.  measuring the time
until the correct PV pops out has problems, as does searching to a fixed depth.
Unless you simply search to the depth required to find the solution, with each
algorithm.  And even that is not so easy to compare.

You can compare node counts.  Search times to find a solution, or search times
to search to a specific depth.  All three reveal different things about the
program...  relying on one is not something I do.  I try to look at them all
to get a better overall picture of what is happening...

This page took 0.03 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.