Author: Dann Corbit
Date: 14:46:19 02/03/03
Go up one level in this thread
On February 03, 2003 at 17:08:47, Robert Allgeuer wrote:
>Others certainly have already done similar tests: while I was recently
>away for some time, I let my computer at home run a self-play experiment
>with the aim to investigate what the impact of opening books and
>endgame tablebases is on the playing strength of a modern chess program.
>
>Method:
>=======
>Crafty 19.1DC was configured with seven different book and tablebase
>combinations. These seven Crafty versions played a round robin tournament
>with 48 subrounds, altogether 1008 games. All duplicate games were
>removed (which left 972 unique games) and these remaining games were
>used to calculate the final rating list, start ELO was 2500.
>Book learning was enabled in order to avoid too many duplicate games,
>preparatory tests had indicated that disabling book learning resulted in
>too many game repetitions, in particular with those Crafty versions
>playing with small books or against an opponent that had no book at all.
>The book´s learning weights were reset before the tournament start.
>In addition to six different "pure" Crafty configurations, one Crafty
>version played with BookThinker, in order to investigate the impact of
>using a much simpler book move selection mechanism than Crafty´s own
>techniques.
>Time controls were deliberately set not too fast in an attempt to obtain
>stable results (i.e. minimal influence of eventual background processes
>etc.).
>
>Participants and Settings:
>==========================
>
>Crafty_lgbk_egtb: Crafty 19.1DC with all 5 piece tablebases and large
> book (12.8MB) as downloaded from the Crafty ftp site.
>Crafty_lgbk_notb: Crafty 19.1DC with no tablebases and large book (12.8MB)
> as downloaded from the Crafty ftp site.
>Crafty_aobk_egtb: Crafty 19.1DC with all 5 piece tablebases and a medium
> sized book (4.25MB) built from Arturo Ochoa´s 2600_updated.pgn file
> (last updated 2.2.2001) with building parameters 22, 1 and 50%.
>Crafty_aobk_notb: Crafty 19.1DC with no tablebases and a medium sized book
> (4.25MB) built from Arturo Ochoa´s 2600_updated.pgn file (last updated
> 2.2.2001) with building parameters 22, 1 and 50%.
>Crafty_aobt_egtb: Crafty 19.1DC and BookThinker 4.1c combination with all
> 5 piece tablebases and a book (4.4MB) built from Arturo Ochoa´s
> 2600_updated.pgn file (last updated 2.2.2001) with BookMaker 1.0b.
>Crafty_smbk_notb: Crafty 19.1DC with no tablebases and a small book (1.0MB)
> built from medium.pgn as downloaded from the Crafty ftp site with
> building parameters 20, 2 and 50%.
>Crafty_nobk_notb: Crafty 19.1DC with no tablesbases and no book.
>
>All versions with Crafty books played with the standard books.bin file
> built from the start.pgn file as downloaded from the Crafty ftp site.
>
>crafty.rc (example for Crafty_lgbk_egtb):
>hash 96M
>hashp 6M
>cache 32M
>drawscore=0
>log off
>book on
>book random 1
>book width 5
>learn 5
>resign 6
>tbpath=f:\tb
>egtb
>
>Main hash: 96MB
>Pawn hash: 6MB
>Tablebases: all 5, 4 and 3 piece EGTBs (when used), 32MB tablebase cache
>Time Control: 20min+5sec
>Ponder off
>Book leaning on
>
>
>Platform and Tools:
>===================
>
>Athlon Thunderbird 1.1MHz
>512 MB RAM
>Windows 2000
>
>Crafty 19.1DC
>BookThinker 4.1c
>BookMaker 1.0b
>Elostat 1.1b
>PGN-Extract 15.0
>Winboard 4.2.3
>WB Tourney Manager 0.60 (Jori Ostrovskij)
>PGN-Sammler
>
>
>Results:
>========
>
> Program Elo + - Games Score Av.Op. Draws
>
> 1 Crafty_lgbk_egtb: 2517 40 23 281 52.8 % 2497 53.0 %
> 2 Crafty_aobt_egtb: 2510 40 23 287 51.7 % 2498 51.9 %
> 3 Crafty_smbk_notb: 2507 42 24 270 51.1 % 2499 51.1 %
> 4 Crafty_lgbk_notb: 2507 41 22 277 51.1 % 2499 58.1 %
> 5 Crafty_aobk_egtb: 2490 22 40 283 48.4 % 2501 55.1 %
> 6 Crafty_aobk_notb: 2486 24 41 271 47.6 % 2502 51.7 %
> 7 Crafty_nobk_notb: 2482 23 40 275 47.1 % 2503 53.5 %
>
>From this the following can be concluded:
>
>The increase of playing strength due to a complete set of 5 piece EGTBs is
>with 95% confidence equal or less than 67 ELO points.
I don't see why you would bother with the number 67. You may as well pick any
number. With NO EGTB, Crafty_smbk_notb and Crafty_lgbk_notb were within 10 ELO
of the best finisher: Crafty_lgbk_egtb. You are missing Crafty_smbk_egtb and so
we cannot conjecture about the difference between having/not-having egtb for the
small book.
> The results suggest
>that the increase is even significantly smaller than that, if not almost
>negligable.
This would be my best guess. However, the uncertainty in the data is enough
that I think any sort of conclusion is questionable.
>The increase of playing strength due to an untuned opening book is with
>95% confidence equal or less than 105 ELO points. The results suggest that
>the increase is even significantly smaller than that.
consider the difference betweeb Crafty_aobk_notb and Crafty_nobk_notb. It seems
to be 4 ELO. If we repeat the experiment, will it be 4 elo again? Or 140? Or
something else?
I don't think we can draw any conclusions.
>There is no measurable difference in playing strength between the Crafty
>version using BookThinker and the comparable versions using native Crafty
>books.
>
>These results are certainly only valid for chess programs that have good
>endgame knowledge coded into their evaluation and whose evaluation function
>allows them to make sensible moves during the opening. If this is not the
>case differences will be bigger. Likewise, tuning a book specifically for
>the playing style of an engine will very probably yield a bigger increase
>in playing strength as perceived here.
>
>Of course - as always - more games would have helped to get more exact
>results. This test also only looks at the influence on playing strength
>against computer opponents.
>
>I would be very interested to have comments, interpretations etc. from
>people who know more about chess programs than I do. I would also be
>interested, whether these results are in line with experience or other
>tests.
Because you had learning enabled, I think the results are very hard to fathom.
The reason I say that is that how a book is interpreted will change over time.
Also, bad lines will be considered in the learning. A version without any
opening book may use this learning data and in essense has not only an opening
book, but also a corrected one.
It looks to me like we might say the following:
It appears unlikely that EGTB files or computer generated opening books will add
several hundred ELO to the crafty chess program when run under an autoplayer at
20min+5sec on {unknown hardware configuration} with 96 MB hash and 6 MB pawn
hash.
It seems likely that an opening book is more valuable than a set of EGTB files
for playing strength, but that is uncertain.
Your conclusions (since they are ceilings) may be fairly safe. But learning
adds another monkey-wrench.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.