Computer Chess Club Archives




Subject: Re: OK, we can make a test ...

Author: Robert Hyatt

Date: 20:19:48 07/31/01

Go up one level in this thread

On July 31, 2001 at 21:38:58, Dieter Buerssner wrote:

>Until now, I tried to trim the citations in my messages. It takes some editing.
>Nobody seems to care. Why?
>I think, if everybody would do this, we all could read this forum much more
>effectively. If only a minority is doing this, it is just work for this
>minority. I give up ...
>Then - you have stated many times, that engine matches on one computer make not
>much sense. Other people have stated, that it may make sense, and discussed, if
>ponder on or off would be more sensible. So, who is right?
>You more or less say, that the "one computer may get sensible results" fraction
>is wrong. Frank suggests an experiment (on 2 computers). You, from the beginning
>call this experiment invalid.

Here is what I said, precisely:

1.  ponder=off is a poor way to play games.  Because that is an unusual way to
run an engine and it is certainly possible (or even probable) that it is not
nearly so well-tested in that mode since it isn't used that way in serious
games.  As a result, you are using the time-allocation code in a way it was
not well-tested, and perhaps in a way it was not designed to be used.  The only
way to see is to look at the code, look at the games, and study how it uses time
with ponder=on and ponder=off to see if they are comparable.

2. ponder=on makes more sense to me on a single machine, and I test like this
all the time.  Both engines are 100% compute-bound, which means each machine
gets 1/2 the total processor time.  It is like using two machines, with each
being 1/2 as fast as the actual machine.  This might be unfair if you think that
one program plays better on faster hardware than it does on slower hardware,
when compared to the other program.  Again you have to play matches both ways
and see if the results are comparable.

Either way, you take a chance on producing results that are not comparable to
the results produced on two separate machines.  Which begs the question "what
is the point?"

>With the same right, the "one computer may get sensible results" fraction can
>say, that you should show evidence, that engine matches on one computer, with
>ponder off will show no reasonable results. So, they can ask you, to show an
>example, where it really makes a difference. How to solve this question?

I won't take the time to find an example, any more than I will take the time
to carefully tune crafty's time allocation for ponder=off matches.  It is time
that is wasted, and I don't have a lot of time to use in general....

I carefully (above) explained both options.  Either one can produce flawed
results.  I know that crafty runs better on faster hardware.  So ponder=off
is better in that regard, perhaps.  I know that crafty is not very good at
time management with ponder=off, so ponder=on is better in that regard,
definitely.  Crafty generally predicts > 50% of the time correctly, which means
Frank's math is wrong and ponder=off does not make things significantly faster
than ponder=on.

Two machines has _none_ of the above issues.  Scientifically, it is the _right_
way to play games.  Why do you think all the commercial authors have machines
connected doing auto play all day long?  Rather than using each machine to run
both programs at the same time with either ponder=off or on?

Think about it...

>You allways seem to assume, that all chess engines are only developed with
>ponder on in mind. I can say, that this is not true. Yace started with ponder
>off, and later I implemented a rather unsophisticated ponder mode. Yace doesn't
>have the puzzling mode, that Crafty has. The time management was written, while
>Yace had no pondering. I just added very few lines of code (certainly less then
>10), to make the engine use more time when pondering is on. So, perhaps, these
>two engines would be a good test for the question, how pondering affects engine
>strength. Especially, because our approach seems to be very different. I do not
>have 2 (reasonable) computers to play engine matches. Neither do I operate an
>account at some ICS myself. All my testing is with ponder off.

Have you never modified the time allocation code after watching it get into a
bit of time trouble with ponder=on?  Or after seeing it reach a time control
with too much time left on the clock?

How do you play games on the servers?  On or off?

How do you play test games to observe the program?  On or off?

I will bet the answers are yes, on and on... respectively.  Just like the rest
of us.  I started off with no pondering either.  But that lasted 2 weeks.  Since
then, it has played about 1/2 million games on the servers.  _every_ one with
ponder=on.  It has played _no_ games with ponder=off except when I ran the test
to answer this issue a while back...

>You, on the other hand, do not care at all about ponder off (which I totally
>respect). So, this seems to be a well thought out example match.
>Unfortunately, the pages of Volker Pittlik are not available right now. He has
>already done many ponder on/off tests. Sure - they wer on one dual computer. My
>interpretation of his results were nevertheless, it really does not make a
>And, as a last point. Many people seem to have fun running engine matches. Only
>a minor fraction of those will have two computers at hand. Why take them all the

They can run them if they wish.  That isn't the issue. I simply say that the
results they produce won't mimic the results obtained with two machines.

>When sombody sends me Crafty-Yace 1-0 on a single computer and a long time
>control, I try to look into the game. It is interesting information for me, as
>well as the result of a whole match (I am not too interested in games won by my
>program ...).
>With best regards,
>Dieter Brner

I'm the same.  I generally ignore wins except for endings against GM players.
There is a lot to be learned there regardless of the outcome.

This page took 0.03 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.