Author: Brian Richardson
Date: 13:42:33 12/18/02
Go up one level in this thread
On December 18, 2002 at 16:20:16, Robert Hyatt wrote: > >That's not what I was talking about. There are two ways you can compare >two chess algorithms, logically. > >1. time until it finds the solution move with the right score. There is a 3rd and IMHO variation, let's call it 1A) find the right move for *whatever* reason (score). Yes, of course, plain 1 (right score) would be best, but in real games, how much should we care? I have been struggling with this with just plain old WAC300. Is 299 in 10 minutes "better" than 295 in 2 minutes in terms of real game performance? It seems to vary, depending on bullet (no), blitz (maybe), or (fairly long) standard time controls (yes). I agree with Bruce that testing methodologies should be discussed and hopefully developed. My hunch is this is the *real* advantage that the commercial programs have worked out--much better testing. Right now for me, a "good" test is about 2 weeks of play on ICC for Tinker. Yes, I do "quick and dirty" WAC and other benchmark tests, but only to see if something seems broken. Yet, even after playing hundreds of games during 2 weeks, the rating is sometimes random (+/- 50), based on the mix of opponents. Brian > >2. time until it completes a particular search depth. > >If you are playing with search extensions (or de-extensions in the case of >null-move) 2 has an obvious problem. Because search depths don't mean much >when you are trying to find a move. > >If you are playing with search extensions or whatever, it is possible to >greatly change the shape of the search tree, so that time to solution looks >good, but time to depth looks bad, or vice-versa. > >I'm personally not convinced there is _any_ way to _really_ compare two >algorithms except to play them in a long match. Tactically stronger != >stronger overall, in all cases. Tactically weaker != weaker overall, in >all cases. > >I think that for times, _both_ approaches are needed. measuring the time >until the correct PV pops out has problems, as does searching to a fixed depth. >Unless you simply search to the depth required to find the solution, with each >algorithm. And even that is not so easy to compare. > >You can compare node counts. Search times to find a solution, or search times >to search to a specific depth. All three reveal different things about the >program... relying on one is not something I do. I try to look at them all >to get a better overall picture of what is happening...
This page took 0.11 seconds to execute
Last modified: Thu, 07 Jul 11 08:48:38 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.