Computer Chess Club Archives




Subject: Re: Proving something is better

Author: Brian Richardson

Date: 13:42:33 12/18/02

Go up one level in this thread

On December 18, 2002 at 16:20:16, Robert Hyatt wrote:
>That's not what I was talking about.  There are two ways you can compare
>two chess algorithms, logically.
>1.  time until it finds the solution move with the right score.

There is a 3rd and IMHO variation, let's call it 1A) find the right move
for *whatever* reason (score).  Yes, of course, plain 1 (right score) would be
best, but in real games, how much should we care?

I have been struggling with this with just plain old WAC300.
Is 299 in 10 minutes "better" than 295 in 2 minutes in terms of real game
performance?  It seems to vary, depending on bullet (no), blitz (maybe), or
(fairly long) standard time controls (yes).

I agree with Bruce that testing methodologies should be discussed and hopefully
developed.  My hunch is this is the *real* advantage that the commercial
programs have worked out--much better testing.

Right now for me, a "good" test is about 2 weeks of play on ICC for Tinker.
Yes, I do "quick and dirty" WAC and other benchmark tests, but only to see if
something seems broken.  Yet, even after playing hundreds of games during 2
weeks, the rating is sometimes random (+/- 50), based on the mix of opponents.


>2.  time until it completes a particular search depth.
>If you are playing with search extensions (or de-extensions in the case of
>null-move) 2 has an obvious problem.  Because search depths don't mean much
>when you are trying to find a move.
>If you are playing with search extensions or whatever, it is possible to
>greatly change the shape of the search tree, so that time to solution looks
>good, but time to depth looks bad, or vice-versa.
>I'm personally not convinced there is _any_ way to _really_ compare two
>algorithms except to play them in a long match.  Tactically stronger !=
>stronger overall, in all cases.  Tactically weaker != weaker overall, in
>all cases.
>I think that for times, _both_ approaches are needed.  measuring the time
>until the correct PV pops out has problems, as does searching to a fixed depth.
>Unless you simply search to the depth required to find the solution, with each
>algorithm.  And even that is not so easy to compare.
>You can compare node counts.  Search times to find a solution, or search times
>to search to a specific depth.  All three reveal different things about the
>program...  relying on one is not something I do.  I try to look at them all
>to get a better overall picture of what is happening...

This page took 0 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.