Author: Robert Allgeuer
Date: 11:38:38 01/07/04
Crafty Generated Book Test ========================== I have used my recently posted Blitz rating list (posted in Winboard forum, http://f11.parsimony.net/forum16635/messages/58889.htm) as 'infrastructure' for a test aiming to measure how generated books (as opposed to hand-tuned books) impact the playing strength of a chess engine. I have already done a similar test in the past (posted in both CCC and Winboard forum, see http://f11.parsimony.net/forum16635/messages/41900.htm), which suggested that the impact on playing strength is quite small (possibly < 30 ELO points, though results were statistically not significant). This previous test was a self-play experiment at a longer time control, while this new test involves matches against a wide range of different engines at Blitz time controls. Method: ======= Crafty 19.6DC has been configured with three different book settings (large book, small book, no book). These three Crafty versions have played against each other (20 games per match) and Gauntlet tournaments (also 20 games per match) against a large and identical set of engines that are listed in my rating list. All duplicate games have been removed and the remaining unique games have been used to calculate a final rating list, hence taking also into account the games between the opponents. Apart from the opening book, the configuration settings of the three Crafty versions were identical. Book learning has been disabled, all three Crafty configurations have played without EGTBs. Participants and Settings: ========================== Crafty v19.06DCntb: Crafty 19.6DC with book created with parameters 50 1 50 from cleanbook.pgn taken from Dann Corbit´s ftp site (size 28.3MB) Crafty v19.06DCntbsbk: Crafty 19.6DC with book created with parameters 20 3 50 from gm2600.pgn taken from the Crafty ftp site (size 564KB) Crafty v19.06DCntbnbk: Crafty 19.6DC with no book. The two configurations with book have played with the standard books.bin file built from the start.pgn file as downloaded from the Crafty ftp site. crafty.rc (example for Crafty v19.06DCntb): hash 96M hashp 8M drawscore=0 log off book on book random 1 book width 5 learn 0 resign 9 no EGTBs for the Crafties; opponents may have used a full set of 5, 4 and 3 piece EGTBs with 8MB TB cache Time Control: 5min + 2sec Ponder off Book leaning off Platform and Tools: =================== Athlon Thunderbird 1.1MHz 512 MB RAM Windows 2000 Crafty 19.6DC (from Dann Corbit´s ftp site) Elostat 1.1b PGN-Extract 15.0 Winboard 4.2.3 WB Tourney Manager 0.60 (Jori Ostrovskij) Results: ======== Program Elo + - Games Score Av.Op. Draws Crafty v19.06DCntb : 2580 21 23 702 63.0 % 2487 31.5 % Crafty v19.06DCntbnbk : 2562 22 23 708 60.7 % 2487 28.4 % Crafty v19.06DCntbsbk : 2540 23 20 714 57.4 % 2488 33.6 % Conclusion: =========== Even with these quite small error margins of close to 20 the differences in playing strength due to different book configurations are still statistically not significant. Nevertheless, these results confirm the results of the previous self-play book test: Both tests suggest that the impact of size and presence of generated books on playing strength seems to be quite small, probably in the order of 30 to 40 ELO points or even below. This result appears to be in line with results that other people have obtained. Hand-tuned books may, however, be a completely different story. If there is a high quality hand-tuned book for Crafty somewhere out there, I would be happy to extend this test and to test such a configuration under the same conditions. Robert Allgeuer
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.