Author: Robert Hyatt
Date: 08:48:53 07/19/00
Go up one level in this thread
On July 19, 2000 at 05:26:14, Mogens Larsen wrote: >On July 18, 2000 at 21:46:42, Robert Hyatt wrote: > >>I don't think that they do implement pondering equally well. But that is the >>native mode the engine operates in. And it is the mode the engine will be in >>when playing serious games. Which means that is the mode that will be used to >>produce what I call important results. So if you have a pondering bug, it will >>show itself in the important games where you use pondering... > >[snip bad analogy] > >Yes, that is true, even though uneven implementation of ponder would also add an >extra degree of freedom to a unstable system of comparing computer chess >programs. Calling it native mode doesn't change that. On a single cpu machine >the native mode is ponder OFF when it comes to engine-engine matches, so let's >keep the semantics out of it. Who makes that rule? I run engine vs engine tests all the time and use ponder=on on a single machine with no problems at all. I use ponder=off only when I am testing/debugging so I can get reproducible results when I need them. If you ask chess program users how they use their engines, 99.999% will say they use it as it comes out of the box... which is "ponder=on". That is why I call it "native mode". That is the _only_ way I play 'serious' games with Crafty on the chess servers, at human events, etc. > Noone disagrees as far as I know with the >importance of playing serious games with the optimal settings achieveable. >Though I believe this is irrelevant for the majority, unless you're blessed with >good hardware or is a good chess player. > >The truth is that your program offer the possibility of disabling ponder, >whereby it becomes a testable feature of your program. It doesn't matter how >well this feature is implemented, because that wouldn't be relevant to ponder >games as well, by your own admission. All I can say, definitively, is that ponder=on has been tested for several million games. Ponder=off has hardly been tested at all. I am happy with the time usage in ponder=on games. I only know that it is tuned for that mode, and that certain assumptions will fail if it is turned off. ponder=off is not intended for playing games, and I suppose I could add that test to the code to not allow it. It is only intended for testing/debugging. Have you ever used the "testev" command? Probably not as that is also a debugging tool I use, but it is useless for anything else... I don't see an obligation for freeware authors _or_ commercial authors to do enough testing to make sure that even oddball configurations work well. Does your automobile float in water? I don't care. Of course if someone drives off a bridge, they might be interested to be sure they have time to get out before it sinks. But I am aware of no testing that requires this to happen. > >The fact that Crafty thinks that it is pondering, while not doing so, is a flaw >in the program. If only flawless programs were tested we wouldn't be able to >test that many programs, if any at all. I suggest applying one of two options: > >1) Disable ponder OFF completely. I can do that. >2) Fix ponder OFF. I don't have the time to play as many games as needed to tune/test this. The current time control code was developed over several years, with lots of tuning and adjusting every few hundred games. > >As long as it exists it's a feature of your program, like it's a feature of many >others. You may ignore the results, but they cannot be described as random. They >reflect the state of the program concerning the features implemented. I don't disagree. I simply state, after every such match I see, "this doesn't necessarily reflect how these two engines would behave in a _real_ chess match under normal circumstances (ponder=on, etc.)" There are other things wrong with one machine matches. Hash table size. EGTB cache size. etc. So you really make _several_ compromises when playing such matches. Sort of like an automobile race with two drivers, but one automobile. It just doesn't show the same thing as a real race. > >Best wishes... >Mogens
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.