Author: Robert Allgeuer
Date: 14:08:47 02/03/03
Others certainly have already done similar tests: while I was recently
away for some time, I let my computer at home run a self-play experiment
with the aim to investigate what the impact of opening books and
endgame tablebases is on the playing strength of a modern chess program.
Method:
=======
Crafty 19.1DC was configured with seven different book and tablebase
combinations. These seven Crafty versions played a round robin tournament
with 48 subrounds, altogether 1008 games. All duplicate games were
removed (which left 972 unique games) and these remaining games were
used to calculate the final rating list, start ELO was 2500.
Book learning was enabled in order to avoid too many duplicate games,
preparatory tests had indicated that disabling book learning resulted in
too many game repetitions, in particular with those Crafty versions
playing with small books or against an opponent that had no book at all.
The book´s learning weights were reset before the tournament start.
In addition to six different "pure" Crafty configurations, one Crafty
version played with BookThinker, in order to investigate the impact of
using a much simpler book move selection mechanism than Crafty´s own
techniques.
Time controls were deliberately set not too fast in an attempt to obtain
stable results (i.e. minimal influence of eventual background processes
etc.).
Participants and Settings:
==========================
Crafty_lgbk_egtb: Crafty 19.1DC with all 5 piece tablebases and large
book (12.8MB) as downloaded from the Crafty ftp site.
Crafty_lgbk_notb: Crafty 19.1DC with no tablebases and large book (12.8MB)
as downloaded from the Crafty ftp site.
Crafty_aobk_egtb: Crafty 19.1DC with all 5 piece tablebases and a medium
sized book (4.25MB) built from Arturo Ochoa´s 2600_updated.pgn file
(last updated 2.2.2001) with building parameters 22, 1 and 50%.
Crafty_aobk_notb: Crafty 19.1DC with no tablebases and a medium sized book
(4.25MB) built from Arturo Ochoa´s 2600_updated.pgn file (last updated
2.2.2001) with building parameters 22, 1 and 50%.
Crafty_aobt_egtb: Crafty 19.1DC and BookThinker 4.1c combination with all
5 piece tablebases and a book (4.4MB) built from Arturo Ochoa´s
2600_updated.pgn file (last updated 2.2.2001) with BookMaker 1.0b.
Crafty_smbk_notb: Crafty 19.1DC with no tablebases and a small book (1.0MB)
built from medium.pgn as downloaded from the Crafty ftp site with
building parameters 20, 2 and 50%.
Crafty_nobk_notb: Crafty 19.1DC with no tablesbases and no book.
All versions with Crafty books played with the standard books.bin file
built from the start.pgn file as downloaded from the Crafty ftp site.
crafty.rc (example for Crafty_lgbk_egtb):
hash 96M
hashp 6M
cache 32M
drawscore=0
log off
book on
book random 1
book width 5
learn 5
resign 6
tbpath=f:\tb
egtb
Main hash: 96MB
Pawn hash: 6MB
Tablebases: all 5, 4 and 3 piece EGTBs (when used), 32MB tablebase cache
Time Control: 20min+5sec
Ponder off
Book leaning on
Platform and Tools:
===================
Athlon Thunderbird 1.1MHz
512 MB RAM
Windows 2000
Crafty 19.1DC
BookThinker 4.1c
BookMaker 1.0b
Elostat 1.1b
PGN-Extract 15.0
Winboard 4.2.3
WB Tourney Manager 0.60 (Jori Ostrovskij)
PGN-Sammler
Results:
========
Program Elo + - Games Score Av.Op. Draws
1 Crafty_lgbk_egtb: 2517 40 23 281 52.8 % 2497 53.0 %
2 Crafty_aobt_egtb: 2510 40 23 287 51.7 % 2498 51.9 %
3 Crafty_smbk_notb: 2507 42 24 270 51.1 % 2499 51.1 %
4 Crafty_lgbk_notb: 2507 41 22 277 51.1 % 2499 58.1 %
5 Crafty_aobk_egtb: 2490 22 40 283 48.4 % 2501 55.1 %
6 Crafty_aobk_notb: 2486 24 41 271 47.6 % 2502 51.7 %
7 Crafty_nobk_notb: 2482 23 40 275 47.1 % 2503 53.5 %
From this the following can be concluded:
The increase of playing strength due to a complete set of 5 piece EGTBs is
with 95% confidence equal or less than 67 ELO points. The results suggest
that the increase is even significantly smaller than that, if not almost
negligable.
The increase of playing strength due to an untuned opening book is with
95% confidence equal or less than 105 ELO points. The results suggest that
the increase is even significantly smaller than that.
There is no measurable difference in playing strength between the Crafty
version using BookThinker and the comparable versions using native Crafty
books.
These results are certainly only valid for chess programs that have good
endgame knowledge coded into their evaluation and whose evaluation function
allows them to make sensible moves during the opening. If this is not the
case differences will be bigger. Likewise, tuning a book specifically for
the playing style of an engine will very probably yield a bigger increase
in playing strength as perceived here.
Of course - as always - more games would have helped to get more exact
results. This test also only looks at the influence on playing strength
against computer opponents.
I would be very interested to have comments, interpretations etc. from
people who know more about chess programs than I do. I would also be
interested, whether these results are in line with experience or other
tests.
Robert
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.