Author: Robert Allgeuer
Date: 13:44:37 03/30/04
I have run a test to quantify further the influence of the opening book on playing strength. This is an extension of the test that I have posted previously (see <http://f11.parsimony.net/forum16635/messages/60135.htm> and also <http://f11.parsimony.net/forum16635/messages/41900.htm> for test at longer time controls). I have used Crafty 19.6 (as compiled by Dann Corbit) for the test; specifically I have also looked at, whether the use of bookc and books files (which essentially make Crafty´s choice of the book move more selective) has a measurable effect on playing strength. Method: ======= Crafty 19.6DC has been configured with five different book settings (combinations of large book, small book, no book; books and bookc present, only books or neither bookc and books). These five Crafty configurations have played against each other (20 games per match) and have performed Gauntlet tournaments (also 20 games per match) against a large and identical set of engines that are listed in my rating list. All duplicate games have been removed and the remaining unique games have been used to calculate a final rating list. Apart from the opening book settings, the configurations of the five Crafty versions were identical. Book learning has been disabled; all three Crafty configurations have played without EGTBs. Participants and Settings: ========================== Crafty v19.06DCntbbookc: Crafty 19.6DC with book created with parameters 50 1 50 from cleanbook.pgn taken from Dann Corbit´s ftp site (size 28.3MB), both bookc and books files present Crafty v19.06DCntb: as above, but only books file present, no bookc file Crafty v19.06DCntbnobookcs: as above but neither bookc nor books file present Crafty v19.06DCntbsbk: Crafty 19.6DC with book created with parameters 20 3 50 from gm2600.pgn taken from the Crafty ftp site (size 564KB), only books present Crafty v19.06DCntbnbk: Crafty 19.6DC with no book at all The files books.bin and bookc.bin are taken from the Crafty ftp site. crafty.rc (example for Crafty v19.06DCntb): hash 96M hashp 8M drawscore=0 log off book on book random 1 book width 5 learn 0 resign 9 no EGTBs for the Crafties; opponents may have used a full set of 5, 4 and 3 piece EGTBs with 8MB TB cache Time Control: 5min + 2sec Ponder off Book leaning off Platform and Tools: =================== Athlon Thunderbird 1.1GHz 512 MB RAM Windows 2000 Crafty 19.6DC (from Dann Corbit´s ftp site) Elostat 1.1b PGN-Extract 15.0 Winboard 4.2.3 WB Tourney Manager 0.60 (Jori Ostrovskij) Results: ======== Program Elo + - Games Score Av.Op. Draws Crafty v19.06DCntbbookc : 2592 20 19 825 61.6 % 2510 36.4 % Crafty v19.06DCntb : 2580 21 20 818 59.7 % 2512 32.8 % Crafty v19.06DCntbnobookcs: 2575 21 19 810 59.3 % 2510 34.1 % Crafty v19.06DCntbnbk : 2568 21 20 824 58.1 % 2511 31.1 % Crafty v19.06DCntbsbk : 2548 22 18 834 55.1 % 2512 35.1 % Conclusion: =========== The influence of bookc and books is small and even after more than 800 games for each configuration statistically not significant. Still, the test results suggest that there may be indeed an increase in playing strength due to the use of a more selective book move selection that favours "engine-friendly" positions, although this increase appears to be quite small. Even the result of Crafty using no book at all is only a little bit worse than the top. Furthermore, when looking at the error margins, even the lowest scoring configuration with the small book based on gm2600.pgn, could in reality be very close to the top. These results confirm the results of my previous book tests and also other tests (e.g. Lousy Lines Tournament of Volker Pittlik): the impact of books (i.e. their presence, their size and the applied move selection strategy) on playing strength seems to be relativly small, probably not more than some 40 ELO points or so. And if an engine has to play without a book altogether, this does not appear to be a huge penalty, either. Robert Allgeuer
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.