Author: Eelco de Groot
Date: 18:57:02 05/09/00
Go up one level in this thread
Hello Mogens, I don't want to restart this whole dicussion and I haven't followed more than half of it but I think some of your criticisms were a little harsh. In practically any experiment there are disturbing influences and I think there were some here too. The biggest influence I could see, one that possibly could have been avoided is that in the beginning some matches were played with booklearning on. If I am mistaken here I hope that somebody can correct me. I know for Rebel that booklearning can be disabled, for Crafty this can be done with the command learn=0. I don't know exactly if those commands can be used in the Hiarcs interface or in winboard for Crafty and if booklearning can be disabled for Fritz 6a too but especially for a repeated Nunn-like test it would be desirable. Okay, I think that is clear. Apart from that I think using the Nunn positions was a good idea from Chessfun, if the object was to see how a. timecontrol or b. pondering on one or two machines affects the strength of an engine combined with use of the timing algorithms involved. I think any not too imbalanced early middlegame position could be used for these experiments if each engine gets to play both colours. In practice of course also the opening books affect the strength of a program (as opposed to engine) but since bookmoves can be played very fast just starting from a Nunn-position does not make much difference for the timing algorithm. The big down side I see in using books and learners is that the books also have a big randomizing effect on the results and secondly if the two learners in a match don't cancel each other out that can mean that the results don't stabilize even after large numbers of games. They are just big noise generators if you want to look at the effect of pondering or timecontrol. Even if you would consider both books and both learners equally good you need much more games to determine differences in engine strength this way. I think it is also possible that a lot of the Rock, Paper, Scissors effects, A beats B beats C beats A that you see in The SSDF results, are mainly caused by the books and learners. They might have a far greater influence in determining results between computers than their analogs opening knowledge and learning during a match for humans. That's just my opinion. Comparing tests on one machine with results on two machines is interesting too to see if there is a discernible effect of how limited resources get divided etc. There could have been some influence from using other programs on just the Windows 98 computer but I don't think that can have had much influence on the results. I think it is easy to test how much a program gets slowed down if you use it with other programs running, by looking at times needed to reach a certain plydepth. As a rather extreme case I slowed Rebels engine for ECTool down by about 10 % max. In the arbitrary position after 1.c4 e5 2.Nc3 I got: Rebel Engine for ECTool. (c) Ed Schröder Engine version : REBEL 9/10 Hash table size : 13 Mb Analysis mode : Analyzing next move Refresh interval : 1000 ms Last Move : 2.Nc3 (Black to play) 00:00 07.00 0.05 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 Nc6xd4 Nf3xe5 Bf8-c5 00:02 08.00 0.07 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Nc6xd4 00:05 09.00 0.07 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Nc6xd4 00:12 10.00 0.10 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Bf8-b4 00:34 11.00 0.10 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Nc6xd4 01:36 12.00 0.08 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Bf8-b4 Now while connecting with server, loading Internet Explorer 5 in Windows 98, loading startpage CCC, reading some messages, going to Gambitsoft, reading some messages and typing in a reply I get for the Rebel engine in ECTool environment: Rebel Engine for ECTool. (c) Ed Schröder Engine version : REBEL 9/10 Hash table size : 13 Mb Analysis mode : Analyzing next move Refresh interval : 1000 ms Last Move : 2.Nc3 (Black to play) 00:00 07.00 0.05 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 Nc6xd4 Nf3xe5 Bf8-c5 00:03 08.00 0.07 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Nc6xd4 00:05 09.00 0.07 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Nc6xd4 00:14 10.00 0.10 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Bf8-b4 00:37 11.00 0.10 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Nc6xd4 01:46 12.00 0.08 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Bf8-b4 03:41 13.00 0.08 Nb8-c6 Ng1-f3 Ng8-f6 d2-d4 e5xd4 Nf3xd4 Bf8-b4 If halving the processor speed costs about 60 elopoints 10% off would be not much more than 8 elopoints difference, just for those moves where the engine had to give in some CPU time. That can't have cost many games, the effect of an unfavourable opening is far greater. 8 elopoints difference if you were to keep up opening browsers, type in messages nonstop for a whole match. I believe I read that the SSDF people who are after all only volunteers used their computers while playing a match too sometimes. I believe I read that they once failed to use the right opening book and the right parameter for activating the learner for Rebel 9 in the past. But let's not get into that now, I just want to say that the SSDF results have had their share of discussions too. Chessfun only recently started with the autoplayer so it's logical that you can run into some unforeseen trouble then. To call her attention for detail appalling I think is really not fair. Which is why I bother explaining my opinions. Regards, Eelco
This page took 0.02 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.