Author: Vasik Rajlich
Date: 01:50:06 02/22/05
Go up one level in this thread
On February 21, 2005 at 16:29:00, Dann Corbit wrote: >On February 21, 2005 at 15:15:49, Tord Romstad wrote: > >>Hi Dann! >> >>On February 21, 2005 at 13:40:37, Dann Corbit wrote: >> >>>This is on the 2.2 GHz 64 bit computer (which had fewer problems than the slower >>>machine). The first set is the raw data, and the second set has had all >>>duplicate games removed. >>> >>> Program Elo + - Games Score Av.Op. Draws >>>1 Polyglot fruit-book : 2553 31 35 372 59.9 % 2483 18.3 % >>>2 Polyglot glaurung-book : 2540 32 35 372 57.5 % 2487 16.7 % >>>3 Polyglot glaurung-no-book: 2481 32 34 372 46.2 % 2507 16.7 % >>>4 Polyglot fruit-no-book : 2427 40 29 372 36.3 % 2525 14.0 % >>> >>> Program Elo + - Games Score Av.Op. Draws >>>1 Polyglot fruit-book : 2550 32 37 348 59.3 % 2484 17.5 % >>>2 Polyglot glaurung-book : 2536 33 35 363 56.7 % 2489 16.5 % >>>3 Polyglot glaurung-no-book: 2483 33 36 343 46.9 % 2505 16.9 % >>>4 Polyglot fruit-no-book : 2430 41 30 354 36.9 % 2524 12.7 % >>> >>>It seems that Fruit may benefit from a book slightly more than glaurung. >> >>This is a big surprise to me. Glaurung has a very primitive development >>eval, and no mobility at all except some very simple stuff for bishops. I >>thought it would be completely unable to find good opening moves on >>its own. >> >>Perhaps the explanation simply is that stronger engines tend to profit >>more from playing with a book, as Sandro has suggested. > >That is highly counter-intuitive to me. >I can understand that strong engines would not be as likely to "get lost" when >playing gambits and things of that nature, but they do not have as much headroom >to benefit from. > >Consider an engine that plays 1000 Elo. Give it a book 100 ply deep with no >errors and all moves of top engine SSDF or SuperGM quality. > >Give the same book to Shredder. > >I cannot imagine how Shredder will benefit more. > >I think (paradoxically) that the opposite is more logical to me. For instance, >the books that the professional programs have will be realistically many orders >of magnitude better than the auto-generated sort that I can produce. And so if >they get a 100 Elo boost, I imagine that a weaker program will get far more from >it because they would have played stupid moves early on. > That's one half of the story. The other half is that the result between two stronger engines will be more closely correlated with the outcome of the opening that the result between two weaker engines. It's hard to say which matters more. >>It could also >>be related to the relative strengths and weaknesses of the two programs. >>When I run test matches between Fruit and Glaurung, I have noticed that >>the Glaurung's scores most of its wins in the middle game, while Fruit >>tends to win most endgames. Perhaps playing without a book reduces >>Fruit's chances of reaching the endgame. > >I also find it surprising, since Fruit has a very sophisicated mobility >calculation. The simplest explanation here is lack of games. I don't recall the confidence on 70 games, but for 1100 games the confidence is 20 rating points. Another thing to keep in mind is that even if you use 95% confidence to draw your conclusions, every 5% of the time, you'll still draw a wrong conclusion. Vas > >>Thanks for doing this experiment, and for your nice choice of engines >>for the experiment! :-) > >I prefer open source engines for my tests (if possible). This test required UCI >because I was also interested in testing the Polyglot book feature at the same >time.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.