Author: Dann Corbit
Date: 21:24:20 07/01/04
Go up one level in this thread
On July 01, 2004 at 11:27:08, Dana Turnmire wrote: >On July 01, 2004 at 09:09:55, francis wrote: > >>if you get computers to generate their own opening books - without using the >>human GM's games as inspiration, then how well do they do? >> >>do they find most of the main lines? > >Try your engine with this test suite. > >BOOK OPENINGS TEST (Larry Kaufman) > >There are many ways to evaluate chess programs. They can be played against each >other, played against human opponents, or tested on problem sets. In this >article I will introduce yet another way to evaluate chess programs. The basic >idea is to compare the program's choice of move to the choices made by masters >over the years. >Here is my procedure: I first selected eight of the most popular openings >from master play. Then, using a large database of master games, I followed each >opening down the path seen most frequently at each move in the database. The >line was terminated when the sample of remaining games dropped below one >hundred. Next, each move in each line which was played at least 75% of the time >but was not an obviously forced recapture was presumed to be the best move at >that point and was marked as a problem. This produced a set of exactly one >hundred problems from opening play in which there exists a best move in the >opinion of at least 75% of masters reaching those exact positions in their own >games. >The method of testing a program is simple. First, turn off its opening book >and put it on analyze mode, or else on infinite level and monitor mode, which is >pretty much the same thing. Then, play through the opening line until a problem >position is reached, let the program think two minutes and record whether or not >the correct move is played, enter the correct move and continue to enter moves >until the next problem in the line is encountered, and repeat the procedure. >When each opening line is completed, reset for new game and go to the next >opening line. The score is simply the number of correct moves out of the one >hundred total. >The eight openings I selected were the Ruy Lopez, the Sicilian with 2 Nf3 d6, >the Winawer French, the Caro-Kann with 3 Nc3, the Slav defense to the Queen's >Gambit, the Nimzoindian defense, the King's Indian defense, and the Gruenfeld >defense. >Of course it must be remembered that this test deals only with how well a >program plays the opening without benefit of an opening book. It may be argued >that since programs do in fact play with opening books this ability is not >crucial in actual practice. On the other hand, the ability to find the best >opening moves without a book is an excelent indication of the overall strategic >and tactical abilities of the program. >The lines tested were as follows (moves in parenthesis are positions to be >tested by the program). > >In the Ruy Lopez: 1 e4 e5 2Nf3 Nc6 3 Bb5 (a6) 4 (Ba4) (Nf6) 5 (0-0) Be7 6 >(Re1) (b5) 7 Bb3 0-0 8 (c3) (d6) 9 (h3) Na5 10 (Bc2) (c5) 11 (d4) Qc7 12 (Nbd2) >cd 13 cd Nc6 14 Nb3 (a5) 15 (Be3) (a4) 16 (Nbd2). > >In the Sicilian: 1 e4 c5 2 Nf3 d6 3 (d4) (cd) 4 (Nd4) (Nf6) 5 (Nc3) a6 6 Bg5 >(e6) 7 (f4) Be7 8 (Qf3) (Qc7) 9 (0-0-0) (Nbd7) 10 g4 (b5) 11 (Bf6) (Nf6) 12 (g5) >Nd7 13 f5 Nc5 14 (f6). > >In the Caro Kann: 1 e4 c6 2 d4 d5 3 Nc3 (de4) 4 Ne4 Bf5 5 (Ng3) (Bg6) 6 h4 >(h6) 7 (Nf3) (Nd7) 8 (h5) Bh7 9 (Bd3) (Bxd3) 10 (Qxd3) Qc7 11 (Bd2) e6 12 >(0-0-0) Ngf6 13 Ne4 0-0-0 14 (g3) Ne4. > >In the French: 1 e4 e6 2 d4 d5 3 Nc3 Bb4 4 e5 c5 5 (a3) (Bc3) 6 bc3 (Ne7) 7 >Qg4 Qc7 8 (Qg7) Rg8 9 Qh7 cd 10 (Ne2) (Nbc6) 11 (f4) (Bd7) 12 (Qd3) dc 13 Nc3 >(a6). > >In the Slav: 1 d4 d5 2 c4 c6 3 Nf3 (Nf6) 4 Nc3 dc4 5 (a4) (Bf5) 6 e3 (e6) 7 >Bc4 (Bb4) 8 (0-0) Nbd7 9 Qe2 0-0 10 e4 (Bg6) 11 (Bd3). > >In the King's Indian: 1 d4 Nf6 2 c4 g6 3 Nc3 Bg7 4 (e4) (d6) 5 Nf3 (0-0) 6 >(Be2) (e5) 7 0-0 Nc6 8 (d5) (Ne7) 9 Ne1 (Nd7) 10 Nd3 (f5) 11 (Bd2) Nf6 12 (f3) >f4 13 c5 (g5) 14 Rc1. > >In the Nimzo-Indian: 1 d4 Nf6 2 c4 e6 3 Nc3 Bb4 4 Qc2 0-0 5 a3 (Bc3) 6 (Qc3) >(b6) 7 (Bg5) (Bb7) 8 f3 h6 9 (Bh4) (d5) 10 (e3) (Nbd7). > >In the Grunfeld: 1 d4 Nf6 2 c4 g6 3 Nc3 d5 4 cd Nd5 5 (e4) (Nc3) 6 bc3 (Bg7) >7 Bc4 c5 8 (Ne2) Nc6 9 (Be3) (0-0) 10 (0-0) Bg4 11 (f3) (Na5) 12 Bd3 (cd) 13 >(cd) (Be6) 14 Rc1 (Ba2) 15 (Qa4) (Be6). Interesting set of problems. I am going to run each of the 100 distinct positions above (regardless of selection via parenthesis) for one hour each using Shredder 7.04 UCI on a 64 bit AMD machine to see how well the moves are chosen. I wonder if any novelties will be discovered.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.