Author: Dann Corbit
Date: 15:48:31 01/14/00
Go up one level in this thread
On January 14, 2000 at 15:46:32, James A. Tackett wrote: > The standard way to test chess programs, i.e., have them play a "regualr >match" against each other does provide interesting information about their >relative strength and playing style. However, the opening books of the programs >and the relatively short matches tend to add noise to the final result and may >distort the truth. How many times have we seen the opening book of a major >program be set up to take advantage of the weaknesses of a competitor? This may >be good for marketing the new program, but it is bad for us consumers. I want to >know who has the strongest playing program, not the slickest, deal from the >bottom of the deck, opening book! > > I do my testing by picking 20 early middlegame positions from grandmaster >play (e.g., Dragon, Ruy, King's Indian, Queen's Gambit etc.) and have the >programs play two games (one White, one Black) from these set positions. A >total of 40 games with opening books turned off! You cannot run a big >tournament this way due to the time requirement, but to decide if program A is >better than program B, this system is very good. But the real programs have the opening books, so it won't tell you which program can beat the others. What that test tells us (exactly) is: "Given this set of starting positions, which program has the upper hand?" And it will take several hundred matches to begin to know the answer. Your basic idea is often tried, though, and it is a good one. It does tell us how programs react in certain situations. Typically, the Nunn-test positions are used. Since these are so well analyzed, it would probably be good to start using a different set now.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.