Author: Ed Schröder
Date: 07:19:48 08/29/98
Go up one level in this thread
I>>Bob is right here. Testing chess program without Permanent Brain (PB) is a >>waste of time if you want an accurate result. The time algorithm's in chess >>programs are written in cooperation with the PB. Without the PB a chess >>program after some time (say after move 20) will automatically become into >>time trouble resulting in bad moves. >I am listening closely and still taking bets on the outcome of my 40/120 match >Crafty 15.17 on PII-400 vs. Junior 5 on P233MMX ... This time with permanent >brain enabled, again starting from the 10 Nunn positions with reversed colors >(i.e. 20 games total). That's the way to do it, program vs program on 2 computers. >This is by the way a nice experiment to verify the effect of PB on/off >setting, I disagree. The system "2 programs on 1 computer" is flawed for reasons I pointed out in my previous posting. Whatever the outcome of your second match you can not compare them. >although I fear that if Crafty wins I will only get the "I told you so" >statements from Bob and if it loses he will advise me to spend 2 weeks of my >life trying to reproduce the result on ICC (thus winning his claim by default >since I will not be able to comply ...). >>Playing games without the PB I consider the same as playing games >>without an opening book. As the opening book the permanent brain >>is an essential part of a chess program. >>- Ed - >Playing games from a set of opening positions (without book) is useful as it >gives you an essentially non-quantitative means of comparing engine >performance in well defined thematical testbeds. I.e. if I want to find out if >and how you >improved Rebel 10 beyond Rebel 9, I will have them play a number of opponents >on >the Nunn positions and (this is important!) not just add up results (this >would >be utterly wrong in my opinion since the initial set of positions might be >biased in some ways) but take a closer look at deviations and understanding of >different types of games (although the Nunn tests usually requires that the >winner takes the initiative and therefore might favour some programs above >others, e.g. Rebel 10 above Rebel 9 ;-)). The same kind of comparison as for >different releases of one program is of course also possible with different >hardware setups, time controls, ... >In fact, "your own" Jeroen proposed a test set of 50 initial positions, so the >basic idea has at least one very important fan even among opening book authors >:-) >I'm pondering whether I should set up a "Nunn site" on my homepage. I will >probably do so as soon as I have results for at least a couple of program >pairings at action chess, then iterating testing with regular Standard time >controls. >One nice side effect is that by using the Nunn positions, nobody can "forget" >to enable books or avoid to give *ALL* games played (wins, losses and draws >alike) >if somebody claims certain results. Since the test has even been kind-of >endorsed by Ossi Weiner himself, it's better to use the "nunnery" than to cook >up something new and face the old hostile accusations again. That's why I am against such systems. First you by-pass the opening book. Next it's quite easy to tune on these positions. Only way out, pick 10-20 new positions, Nunn alike at least every year. Serious suggestion (with a smiley) Pick 10-20 positions using Rebel's "shuffle book". You entirely get random book positions each time. Skip won/lost opening lines. Keep the balanced ones. Use an unknown opening book for this. Rebel can convert Fritz, Genius and Chessmaster books. Mix them, shake them and call it the "Berger" test :-) - Ed -
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.