Author: Robert Hyatt
Date: 20:19:48 07/31/01
Go up one level in this thread
On July 31, 2001 at 21:38:58, Dieter Buerssner wrote: > >Until now, I tried to trim the citations in my messages. It takes some editing. >Nobody seems to care. Why? > >I think, if everybody would do this, we all could read this forum much more >effectively. If only a minority is doing this, it is just work for this >minority. I give up ... > >Then - you have stated many times, that engine matches on one computer make not >much sense. Other people have stated, that it may make sense, and discussed, if >ponder on or off would be more sensible. So, who is right? > >You more or less say, that the "one computer may get sensible results" fraction >is wrong. Frank suggests an experiment (on 2 computers). You, from the beginning >call this experiment invalid. Here is what I said, precisely: 1. ponder=off is a poor way to play games. Because that is an unusual way to run an engine and it is certainly possible (or even probable) that it is not nearly so well-tested in that mode since it isn't used that way in serious games. As a result, you are using the time-allocation code in a way it was not well-tested, and perhaps in a way it was not designed to be used. The only way to see is to look at the code, look at the games, and study how it uses time with ponder=on and ponder=off to see if they are comparable. 2. ponder=on makes more sense to me on a single machine, and I test like this all the time. Both engines are 100% compute-bound, which means each machine gets 1/2 the total processor time. It is like using two machines, with each being 1/2 as fast as the actual machine. This might be unfair if you think that one program plays better on faster hardware than it does on slower hardware, when compared to the other program. Again you have to play matches both ways and see if the results are comparable. Either way, you take a chance on producing results that are not comparable to the results produced on two separate machines. Which begs the question "what is the point?" > >With the same right, the "one computer may get sensible results" fraction can >say, that you should show evidence, that engine matches on one computer, with >ponder off will show no reasonable results. So, they can ask you, to show an >example, where it really makes a difference. How to solve this question? I won't take the time to find an example, any more than I will take the time to carefully tune crafty's time allocation for ponder=off matches. It is time that is wasted, and I don't have a lot of time to use in general.... I carefully (above) explained both options. Either one can produce flawed results. I know that crafty runs better on faster hardware. So ponder=off is better in that regard, perhaps. I know that crafty is not very good at time management with ponder=off, so ponder=on is better in that regard, definitely. Crafty generally predicts > 50% of the time correctly, which means Frank's math is wrong and ponder=off does not make things significantly faster than ponder=on. Two machines has _none_ of the above issues. Scientifically, it is the _right_ way to play games. Why do you think all the commercial authors have machines connected doing auto play all day long? Rather than using each machine to run both programs at the same time with either ponder=off or on? Think about it... > >You allways seem to assume, that all chess engines are only developed with >ponder on in mind. I can say, that this is not true. Yace started with ponder >off, and later I implemented a rather unsophisticated ponder mode. Yace doesn't >have the puzzling mode, that Crafty has. The time management was written, while >Yace had no pondering. I just added very few lines of code (certainly less then >10), to make the engine use more time when pondering is on. So, perhaps, these >two engines would be a good test for the question, how pondering affects engine >strength. Especially, because our approach seems to be very different. I do not >have 2 (reasonable) computers to play engine matches. Neither do I operate an >account at some ICS myself. All my testing is with ponder off. > Have you never modified the time allocation code after watching it get into a bit of time trouble with ponder=on? Or after seeing it reach a time control with too much time left on the clock? How do you play games on the servers? On or off? How do you play test games to observe the program? On or off? I will bet the answers are yes, on and on... respectively. Just like the rest of us. I started off with no pondering either. But that lasted 2 weeks. Since then, it has played about 1/2 million games on the servers. _every_ one with ponder=on. It has played _no_ games with ponder=off except when I ran the test to answer this issue a while back... >You, on the other hand, do not care at all about ponder off (which I totally >respect). So, this seems to be a well thought out example match. > >Unfortunately, the pages of Volker Pittlik are not available right now. He has >already done many ponder on/off tests. Sure - they wer on one dual computer. My >interpretation of his results were nevertheless, it really does not make a >difference. > >And, as a last point. Many people seem to have fun running engine matches. Only >a minor fraction of those will have two computers at hand. Why take them all the >fun? They can run them if they wish. That isn't the issue. I simply say that the results they produce won't mimic the results obtained with two machines. > >When sombody sends me Crafty-Yace 1-0 on a single computer and a long time >control, I try to look into the game. It is interesting information for me, as >well as the result of a whole match (I am not too interested in games won by my >program ...). > >With best regards, >Dieter Bürßner I'm the same. I generally ignore wins except for endings against GM players. There is a lot to be learned there regardless of the outcome.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.