Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: OK, we can make a test ...

Author: Uri Blass

Date: 07:11:58 08/01/01

Go up one level in this thread


On July 31, 2001 at 23:19:48, Robert Hyatt wrote:

>On July 31, 2001 at 21:38:58, Dieter Buerssner wrote:
>
>>
>>Until now, I tried to trim the citations in my messages. It takes some editing.
>>Nobody seems to care. Why?
>>
>>I think, if everybody would do this, we all could read this forum much more
>>effectively. If only a minority is doing this, it is just work for this
>>minority. I give up ...
>>
>>Then - you have stated many times, that engine matches on one computer make not
>>much sense. Other people have stated, that it may make sense, and discussed, if
>>ponder on or off would be more sensible. So, who is right?
>>
>>You more or less say, that the "one computer may get sensible results" fraction
>>is wrong. Frank suggests an experiment (on 2 computers). You, from the beginning
>>call this experiment invalid.
>
>
>
>Here is what I said, precisely:
>
>1.  ponder=off is a poor way to play games.  Because that is an unusual way to
>run an engine and it is certainly possible (or even probable) that it is not
>nearly so well-tested in that mode since it isn't used that way in serious
>games.  As a result, you are using the time-allocation code in a way it was
>not well-tested, and perhaps in a way it was not designed to be used.  The only
>way to see is to look at the code, look at the games, and study how it uses time
>with ponder=on and ponder=off to see if they are comparable.
>
>2. ponder=on makes more sense to me on a single machine, and I test like this
>all the time.  Both engines are 100% compute-bound, which means each machine
>gets 1/2 the total processor time.  It is like using two machines, with each
>being 1/2 as fast as the actual machine.  This might be unfair if you think that
>one program plays better on faster hardware than it does on slower hardware,
>when compared to the other program.  Again you have to play matches both ways
>and see if the results are comparable.
>
>Either way, you take a chance on producing results that are not comparable to
>the results produced on two separate machines.  Which begs the question "what
>is the point?"
>
>
>
>
>
>
>
>
>
>>
>>With the same right, the "one computer may get sensible results" fraction can
>>say, that you should show evidence, that engine matches on one computer, with
>>ponder off will show no reasonable results. So, they can ask you, to show an
>>example, where it really makes a difference. How to solve this question?
>
>I won't take the time to find an example, any more than I will take the time
>to carefully tune crafty's time allocation for ponder=off matches.  It is time
>that is wasted, and I don't have a lot of time to use in general....
>
>I carefully (above) explained both options.  Either one can produce flawed
>results.  I know that crafty runs better on faster hardware.  So ponder=off
>is better in that regard, perhaps.  I know that crafty is not very good at
>time management with ponder=off, so ponder=on is better in that regard,
>definitely.  Crafty generally predicts > 50% of the time correctly, which means
>Frank's math is wrong and ponder=off does not make things significantly faster
>than ponder=on.
>
>Two machines has _none_ of the above issues.  Scientifically, it is the _right_
>way to play games.  Why do you think all the commercial authors have machines
>connected doing auto play all day long?  Rather than using each machine to run
>both programs at the same time with either ponder=off or on?
>
>Think about it...
>
>
>
>
>
>>
>>You allways seem to assume, that all chess engines are only developed with
>>ponder on in mind. I can say, that this is not true. Yace started with ponder
>>off, and later I implemented a rather unsophisticated ponder mode. Yace doesn't
>>have the puzzling mode, that Crafty has. The time management was written, while
>>Yace had no pondering. I just added very few lines of code (certainly less then
>>10), to make the engine use more time when pondering is on. So, perhaps, these
>>two engines would be a good test for the question, how pondering affects engine
>>strength. Especially, because our approach seems to be very different. I do not
>>have 2 (reasonable) computers to play engine matches. Neither do I operate an
>>account at some ICS myself. All my testing is with ponder off.
>>
>
>
>Have you never modified the time allocation code after watching it get into a
>bit of time trouble with ponder=on?  Or after seeing it reach a time control
>with too much time left on the clock?
>
>How do you play games on the servers?  On or off?
>
>How do you play test games to observe the program?  On or off?
>
>I will bet the answers are yes, on and on... respectively.

I guess that playing test games to observe yace are done usually with ponder
off.

Dieter explained that Fritz get most of the machine time if he tries to play
engine-engine games against Fritz with ponder on  so I guess that he plays these
games with ponder off.

Uri



This page took 0.03 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.