Author: Don Dailey
Date: 09:17:17 04/30/98
Go up one level in this thread
On April 30, 1998 at 04:16:05, Alessandro Morales wrote: >Book is important because the program is unable to understand the >chosen positions if they don't meet its play style. >But for me the most important rules to compile a rating list are: >1) Same hardware (processor and RAM) for all programs. >2) Each program must play the same number of games against each other >opponent. > >Alessandro Morales You raise important issues here. I agree with all of your points except the "same hardware" issue. I definitely think the ratings are affected by the choice of opponents. There should be a system where opponents are chosen by a standardized method. It might not make sense to require the weakest programs to play against the strongest, but the choice should not be arbitrary either. The best should definitely have to play the same number of games among themselves. One possible system is for these games to be arranged as consecutive tournaments, each player will find his (or it's) own level this way. Each round could be a 10 game match, etc. As far as the hardware is concerned, I believe each program has a right to optimize it's play. For instance a program needing much RAM should not be deprived of it. In general, really fast searchers will tend to need bigger hash tables since they can fill them more rapidly. There should be no artificial penalty for being fast. Also, a program may be designed to run particularly well on one hardware platform over another one. Again, this should be reflected in the testing. But to make this fair, the hardware and platform should be fully specified. It should be clear that program X got a rating or R running on some specified platform with so much RAM etc. It's probably good to run each program with 2 or 3 configurations also so that pepole can see how they are affected by the differences. I think it would be nice to have 3 or 4 popular configurations defined in advance, run each program on them and also let the programmer be accomodated with his own requests when this is at all possible. Suppose I came up with some super-duper chess algorithm that was highly dependant on having huge amounts of memory available, otherwise it was worthless? Wouldn't it be useful to be able to benchmark this configuration? To make sure no one was mislead, the complete configuration should be specified. As far as book issues, I'm very undecided on this. A part of me wants to know how strong the engine is, not a memorized book. But another part of me says that even with human players, the book is an integral part of the game. I'm leaning in this direction because who says search is the only criteria for selecting a move? If new technology appeared that allowed us to construct 32 piece (solving the game) databases and this was used as the basis for an omnicient chess program, I would applaud the development and not call it trickery. I like the model of viewing a chess program as a "black box" and not concerning ourselves with exactly how the moves are selected. - Don
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.