Author: Dana Turnmire
Date: 09:04:42 05/11/01
Go up one level in this thread
On May 11, 2001 at 11:49:16, Miguel A. Ballicora wrote: >On May 11, 2001 at 06:17:13, Dana Turnmire wrote: > >>BOOK OPENINGS TEST (Larry Kaufman) >> >>There are many ways to evaluate chess programs. They can be played against each >>other, played against human opponents, or tested on problem sets. In this >>article I will introduce yet another way to evaluate chess programs. The basic >>idea is to compare the program's choice of move to the choices made by masters >>over the years. >> Here is my procedure: I first selected eight of the most popular openings >>from master play. Then, using a large database of master games, I followed each >>opening down the path seen most frequently at each move in the database. The >>line was terminated when the sample of remaining games dropped below one >>hundred. Next, each move in each line which was played at least 75% of the time >>but was not an obviously forced recapture was presumed to be the best move at >>that point and was marked as a problem. This produced a set of exactly one >>hundred problems from opening play in which there exists a best move in the >>opinion of at least 75% of masters reaching those exact positions in their own >>games. >> The method of testing a program is simple. First, turn off its opening book >>and put it on analyze mode, or else on infinite level and monitor mode, which is >>pretty much the same thing. Then, play through the opening line until a problem >>position is reached, let the program think two minutes and record whether or not >>the correct move is played, enter the correct move and continue to enter moves >>until the next problem in the line is encountered, and repeat the procedure. >>When each opening line is completed, reset for new game and go to the next >>opening line. The score is simply the number of correct moves out of the one >>hundred total. >> The eight openings I selected were the Ruy Lopez, the Sicilian with 2 Nf3 d6, >>the Winawer French, the Caro-Kann with 3 Nc3, the Slav defense to the Queen's >>Gambit, the Nimzoindian defense, the King's Indian defense, and the Gruenfeld >>defense. >> Of course it must be remembered that this test deals only with how well a >>program plays the opening without benefit of an opening book. It may be argued >>that since programs do in fact play with opening books this ability is not >>crucial in actual practice. On the other hand, the ability to find the best >>opening moves without a book is an excelent indication of the overall strategic >>and tactical abilities of the program. >> The lines tested were as follows (moves in parenthesis are positions to be >>tested by the program). >> >> In the Ruy Lopez: 1 e4 e5 2Nf3 Nc6 3 Bb5 (a6) 4 (Ba4) (Nf6) 5 (0-0) Be7 6 >>(Re1) (b5) 7 Bb3 0-0 8 (c3) (d6) 9 (h3) Na5 10 (Bc2) (c5) 11 (d4) Qc7 12 (Nbd2) >>cd 13 cd Nc6 14 Nb3 (a5) 15 (Be3) (a4) 16 (Nbd2). >> >> In the Sicilian: 1 e4 c5 2 Nf3 d6 3 (d4) (cd) 4 (Nd4) (Nf6) 5 (Nc3) a6 6 Bg5 >>(e6) 7 (f4) Be7 8 (Qf3) (Qc7) 9 (0-0-0) (Nbd7) 10 g4 (b5) 11 (Bf6) (Nf6) 12 (g5) >>Nd7 13 f5 Nc5 14 (f6). >> >> In the Caro Kann: 1 e4 c6 2 d4 d5 3 Nc3 (de4) 4 Ne4 Bf5 5 (Ng3) (Bg6) 6 h4 >>(h6) 7 (Nf3) (Nd7) 8 (h5) Bh7 9 (Bd3) (Bxd3) 10 (Qxd3) Qc7 11 (Bd2) e6 12 >>(0-0-0) Ngf6 13 Ne4 0-0-0 14 (g3) Ne4. >> >> In the French: 1 e4 e6 2 d4 d5 3 Nc3 Bb4 4 e5 c5 5 (a3) (Bc3) 6 bc3 (Ne7) 7 >>Qg4 Qc7 8 (Qg7) Rg8 9 Qh7 cd 10 (Ne2) (Nbc6) 11 (f4) (Bd7) 12 (Qd3) dc 13 Nc3 >>(a6). >> >> In the Slav: 1 d4 d5 2 c4 c6 3 Nf3 (Nf6) 4 Nc3 dc4 5 (a4) (Bf5) 6 e3 (e6) 7 >>Bc4 (Bb4) 8 (0-0) Nbd7 9 Qe2 0-0 10 e4 (Bg6) 11 (Bd3). >> >> In the King's Indian: 1 d4 Nf6 2 c4 g6 3 Nc3 Bg7 4 (e4) (d6) 5 Nf3 (0-0) 6 >>(Be2) (e5) 7 0-0 Nc6 8 (d5) (Ne7) 9 Ne1 (Nd7) 10 Nd3 (f5) 11 (Bd2) Nf6 12 (f3) >>f4 13 c5 (g5) 14 Rc1. >> >> In the Nimzo-Indian: 1 d4 Nf6 2 c4 e6 3 Nc3 Bb4 4 Qc2 0-0 5 a3 (Bc3) 6 (Qc3) >>(b6) 7 (Bg5) (Bb7) 8 f3 h6 9 (Bh4) (d5) 10 (e3) (Nbd7). >> >> In the Grunfeld: 1 d4 Nf6 2 c4 g6 3 Nc3 d5 4 cd Nd5 5 (e4) (Nc3) 6 bc3 (Bg7) >>7 Bc4 c5 8 (Ne2) Nc6 9 (Be3) (0-0) 10 (0-0) Bg4 11 (f3) (Na5) 12 Bd3 (cd) 13 >>(cd) (Be6) 14 Rc1 (Ba2) 15 (Qa4) (Be6). > >This is interesting to test just for fun, but it means little or nothing. >There are several moves that could be replaced by others that could be equally >good or they just transpose. Who is going to explain to Kramnik that >Nf6 (Ruy Lopez Berlin) is inferior than a6? >6.... Ne7 in the french is ridiculous. Qc7 is almost the same and transposes >after Qg4. >In the Caro-Kann, Nf6 can replace Bxd3 perfectly since transposes to main >variation (this trick has been used by Larsen since he wanted to deviate >at a proper time). >In the slav, e6 will go to the Meran systems and the like rather than Nf6 >What's wrong with that? >Grunfeld, nothing wrong with 8.Be3, it transposes. >etc. etc. etc. >The idea of the test is interesting, but it should be weed out of some >of the moves that are supposed to be "the move of choice". > >Regards, >Miguel I believe you are right. There would have to be a definite "best" move to really be effective as a solid test.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.