Author: Kolss
Date: 06:09:22 07/02/04
Go up one level in this thread
On July 01, 2004 at 11:27:08, Dana Turnmire wrote:
>On July 01, 2004 at 09:09:55, francis wrote:
>
>>if you get computers to generate their own opening books - without using the
>>human GM's games as inspiration, then how well do they do?
>>
>>do they find most of the main lines?
>
>Try your engine with this test suite.
>
>BOOK OPENINGS TEST (Larry Kaufman)
>
>There are many ways to evaluate chess programs. They can be played against each
>other, played against human opponents, or tested on problem sets. In this
>article I will introduce yet another way to evaluate chess programs. The basic
>idea is to compare the program's choice of move to the choices made by masters
>over the years.
>Here is my procedure: I first selected eight of the most popular openings
>from master play. Then, using a large database of master games, I followed each
>opening down the path seen most frequently at each move in the database. The
>line was terminated when the sample of remaining games dropped below one
>hundred. Next, each move in each line which was played at least 75% of the time
>but was not an obviously forced recapture was presumed to be the best move at
>that point and was marked as a problem. This produced a set of exactly one
>hundred problems from opening play in which there exists a best move in the
>opinion of at least 75% of masters reaching those exact positions in their own
>games.
>The method of testing a program is simple. First, turn off its opening book
>and put it on analyze mode, or else on infinite level and monitor mode, which is
>pretty much the same thing. Then, play through the opening line until a problem
>position is reached, let the program think two minutes and record whether or not
>the correct move is played, enter the correct move and continue to enter moves
>until the next problem in the line is encountered, and repeat the procedure.
>When each opening line is completed, reset for new game and go to the next
>opening line. The score is simply the number of correct moves out of the one
>hundred total.
>The eight openings I selected were the Ruy Lopez, the Sicilian with 2 Nf3 d6,
>the Winawer French, the Caro-Kann with 3 Nc3, the Slav defense to the Queen's
>Gambit, the Nimzoindian defense, the King's Indian defense, and the Gruenfeld
>defense.
>Of course it must be remembered that this test deals only with how well a
>program plays the opening without benefit of an opening book. It may be argued
>that since programs do in fact play with opening books this ability is not
>crucial in actual practice. On the other hand, the ability to find the best
>opening moves without a book is an excelent indication of the overall strategic
>and tactical abilities of the program.
>The lines tested were as follows (moves in parenthesis are positions to be
>tested by the program).
>
>In the Ruy Lopez: 1 e4 e5 2Nf3 Nc6 3 Bb5 (a6) 4 (Ba4) (Nf6) 5 (0-0) Be7 6
>(Re1) (b5) 7 Bb3 0-0 8 (c3) (d6) 9 (h3) Na5 10 (Bc2) (c5) 11 (d4) Qc7 12 (Nbd2)
>cd 13 cd Nc6 14 Nb3 (a5) 15 (Be3) (a4) 16 (Nbd2).
>
>In the Sicilian: 1 e4 c5 2 Nf3 d6 3 (d4) (cd) 4 (Nd4) (Nf6) 5 (Nc3) a6 6 Bg5
>(e6) 7 (f4) Be7 8 (Qf3) (Qc7) 9 (0-0-0) (Nbd7) 10 g4 (b5) 11 (Bf6) (Nf6) 12 (g5)
>Nd7 13 f5 Nc5 14 (f6).
>
>In the Caro Kann: 1 e4 c6 2 d4 d5 3 Nc3 (de4) 4 Ne4 Bf5 5 (Ng3) (Bg6) 6 h4
>(h6) 7 (Nf3) (Nd7) 8 (h5) Bh7 9 (Bd3) (Bxd3) 10 (Qxd3) Qc7 11 (Bd2) e6 12
>(0-0-0) Ngf6 13 Ne4 0-0-0 14 (g3) Ne4.
>
>In the French: 1 e4 e6 2 d4 d5 3 Nc3 Bb4 4 e5 c5 5 (a3) (Bc3) 6 bc3 (Ne7) 7
>Qg4 Qc7 8 (Qg7) Rg8 9 Qh7 cd 10 (Ne2) (Nbc6) 11 (f4) (Bd7) 12 (Qd3) dc 13 Nc3
>(a6).
>
>In the Slav: 1 d4 d5 2 c4 c6 3 Nf3 (Nf6) 4 Nc3 dc4 5 (a4) (Bf5) 6 e3 (e6) 7
>Bc4 (Bb4) 8 (0-0) Nbd7 9 Qe2 0-0 10 e4 (Bg6) 11 (Bd3).
>
>In the King's Indian: 1 d4 Nf6 2 c4 g6 3 Nc3 Bg7 4 (e4) (d6) 5 Nf3 (0-0) 6
>(Be2) (e5) 7 0-0 Nc6 8 (d5) (Ne7) 9 Ne1 (Nd7) 10 Nd3 (f5) 11 (Bd2) Nf6 12 (f3)
>f4 13 c5 (g5) 14 Rc1.
>
>In the Nimzo-Indian: 1 d4 Nf6 2 c4 e6 3 Nc3 Bb4 4 Qc2 0-0 5 a3 (Bc3) 6 (Qc3)
>(b6) 7 (Bg5) (Bb7) 8 f3 h6 9 (Bh4) (d5) 10 (e3) (Nbd7).
>
>In the Grunfeld: 1 d4 Nf6 2 c4 g6 3 Nc3 d5 4 cd Nd5 5 (e4) (Nc3) 6 bc3 (Bg7)
>7 Bc4 c5 8 (Ne2) Nc6 9 (Be3) (0-0) 10 (0-0) Bg4 11 (f3) (Na5) 12 Bd3 (cd) 13
>(cd) (Be6) 14 Rc1 (Ba2) 15 (Qa4) (Be6).
Hi,
Needless to say that I do not always agree with the GM moves... :-)
Anyway, here is the PGN (moves to be tested are marked with "!"):
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "C99"]
{17 moves} 1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 $1 4. Ba4 $1 Nf6 $1 5. O-O $1 Be7 6.
Re1 $1 b5 $1 7. Bb3 O-O 8. c3 $1 d6 $1 9. h3 $1 Na5 10. Bc2 $1 c5 $1 11. d4 $1
Qc7 12. Nbd2 $1 cxd4 13. cxd4 Nc6 14. Nb3 a5 $1 15. Be3 $1 a4 $1 16. Nbd2 $1 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "B99"]
{16 moves} 1. e4 c5 2. Nf3 d6 3. d4 $1 cxd4 $1 4. Nxd4 $1 Nf6 $1 5. Nc3 $1 a6
6. Bg5 e6 $1 7. f4 $1 Be7 8. Qf3 $1 Qc7 $1 9. O-O-O $1 Nbd7 $1 10. g4 b5 $1 11.
Bxf6 $1 Nxf6 $1 12. g5 $1 Nd7 13. f5 Nc5 14. f6 $1 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "B19"]
{13 moves} 1. e4 c6 2. d4 d5 3. Nc3 dxe4 $1 4. Nxe4 Bf5 5. Ng3 $1 Bg6 $1 6. h4
h6 $1 7. Nf3 $1 Nd7 $1 8. h5 $1 Bh7 9. Bd3 $1 Bxd3 $1 10. Qxd3 $1 Qc7 11. Bd2
$1 e6 12. O-O-O $1 Ngf6 13. Ne4 O-O-O 14. g3 $1 Nxe4 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "C19"]
{10 moves} 1. e4 e6 2. d4 d5 3. Nc3 Bb4 4. e5 c5 5. a3 $1 Bxc3+ $1 6. bxc3 Ne7
$1 7. Qg4 Qc7 8. Qxg7 $1 Rg8 9. Qxh7 cxd4 10. Ne2 $1 Nbc6 $1 11. f4 $1 Bd7 $1
12. Qd3 $1 dxc3 13. Nxc3 a6 $1 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "D19"]
{8 moves} 1. d4 d5 2. c4 c6 3. Nf3 Nf6 $1 4. Nc3 dxc4 5. a4 $1 Bf5 $1 6. e3 e6
$1 7. Bxc4 Bb4 $1 8. O-O $1 Nbd7 9. Qe2 O-O 10. e4 Bg6 $1 11. Bd3 $1 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "E98"]
{12 moves} 1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. e4 $1 d6 $1 5. Nf3 O-O $1 6. Be2 $1
e5 $1 7. O-O Nc6 8. d5 $1 Ne7 $1 9. Ne1 Nd7 $1 10. Nd3 f5 $1 11. Bd2 $1 Nf6 12.
f3 $1 f4 13. c5 g5 $1 14. Rc1 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "E32"]
{9 moves} 1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. Qc2 O-O 5. a3 Bxc3+ $1 6. Qxc3 $1 b6
$1 7. Bg5 $1 Bb7 $1 8. f3 h6 9. Bh4 $1 d5 $1 10. e3 $1 Nbd7 $1 *
[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[ECO "D89"]
{15 moves} 1. d4 Nf6 2. c4 g6 3. Nc3 d5 4. cxd5 Nxd5 5. e4 $1 Nxc3 $1 6. bxc3
Bg7 $1 7. Bc4 c5 8. Ne2 $1 Nc6 9. Be3 $1 O-O $1 10. O-O $1 Bg4 11. f3 $1 Na5 $1
12. Bd3 cxd4 $1 13. cxd4 $1 Be6 $1 14. Rc1 Bxa2 $1 15. Qa4 $1 Be6 $1 *
Best regards - Munjong.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.