Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: OK, we can make a test ...

Author: Dieter Buerssner

Date: 18:38:58 07/31/01

Go up one level in this thread


On July 30, 2001 at 21:26:59, Robert Hyatt wrote:

>On July 30, 2001 at 10:44:22, Frank Quisinsky wrote:
>
>>On July 30, 2001 at 09:31:28, Robert Hyatt wrote:
>>
>>>On July 29, 2001 at 10:50:47, Frank Quisinsky wrote:
>>>
>>>>
>>>>Hi Bob,
>>>>
>>>>the same discuss then for 2 years here :-)
>>>>
>>>>I, the ex non ponderer, will say the following:
>>>>
>>>>01.
>>>>You have right if you say that with ponder = off engines have time manangment
>>>>problems, not all and not in all games but it's an important point. I know that
>>>>this is for statistics not sooo important (ELO statistic).
>>>>
>>>>02.
>>>>Match without ponder on single system with Athlon 1Ghz:
>>>>Engine A = 1Ghz
>>>>Engine B = 1Ghz
>>>>
>>>>Match with ponder on a single system with Athlon 1Ghz:
>>>>Engine A = ~ 497-500Mhz
>>>>Engine B = ~ 497-500Mhz
>>>>
>>>>Now we have 25-40% ponder hits if the engines play on the same level (after my
>>>>experiments).
>>>>
>>>>Result:
>>>>500Mhz + 25-40% ponder hits = ~700MHz.
>>>>
>>>>With ponder = on, the matches are running with 700Mhz on 1Ghz Athlon!
>>>>With ponder = off, the matches are running with 1Ghz on 1Ghz Athlon!
>>>>
>>>>I believe better is to play with ponder = off on single system. OK, the time
>>>>managment is a good point so say please with ponder but 300 MHz is a better
>>>>point to say please without ponder.
>>>>
>>>>03.
>>>>4-piece and 5-piece tablebases and engine-engine matches on a Dual system ...
>>>>
>>>>This is not very clear.
>>>>We have engines witch used tablebases very aggressive or not very aggressive.
>>>>
>>>>Example:
>>>>Gromit - Patzer with 5-pieces ...
>>>>
>>>>Gromit played move 50 in 2:45 with ponder = on and 5-pieces.
>>>>Only 20-30% processor time in this game if Gromit used 5-pieces.
>>>>Patzer played move 50 in 1.25 with ponder = on and 5-pieces.
>>>>
>>>>Now the same positions with 4-pieces ...
>>>>
>>>>Gromit play move 50 in 1.58! with ponder = on and *4-pieces*.
>>>>Patzer played move 50 in 1.13 with ponder = on and *4-pieces*.
>>>>
>>>>Without 4-piece tablesbases are the differents not very important.
>>>>
>>>>In my opinion it is better to play eng-eng matches with ponder on a dual system
>>>>with 4-piece tablebases.
>>>>
>>>>Best
>>>>Frank
>>>
>>>
>>>Your math is fine.
>>
>>Hi Bob,
>>
>>good, for two years you say my math is bad, but in this case you have right (I
>>mean for two years) :-))
>>
>>But your reasoning is wrong.  Which would you rather do:
>>
>>>(1) play a match between two programs, using their strongest settings, and using
>>>two 700mhz processors;
>>
>>>(2) play a match between two programs, using weaker settings on at least one, if
>>>not both, using two 1ghz processors?
>>>
>>>
>>>I vote for (1) because of using the strongest settings.  You are voting for (2)
>>>to avoid wasting compute cycles.  I think (1) is more important...
>>
>>Yes, this is a good idea and a nice experiment.
>>I have an Dual Pentium III 733 MHz and an Dual Pentium III 1GHz.
>>
>>But it is not 100%ig clear which test exactly!
>>
>>What do you think about the following match idea:
>>Only with one CPUs to point 1-4.
>>
>>01.
>>Crafty 18.10 on Dual Pentium III 733Mhz with ponder against Yace 0.99.50 on Dual
>>Pentium III 1.05GHz (the machine is running with FSB = 140) without ponder, 40
>>moves in 40 minutes, 50 games.
>>
>>02.
>>Yace 0.99.50 on Dual Pentium III 733Mhz with ponder against Crafty 18.10 on Dual
>>Pentium III 1.05Ghz without ponder, 40 moves in 40 minutes, 50 games.
>>
>>03.
>>The same match but Crafty 18.10 without ponder (1.05Ghz) and Yace 0.99.50
>>(1.05Ghz) without ponder, 50 games.
>>
>>04.
>>After this matches, a new match Crafty 18.10 (1.05Ghz) with ponder - Yace
>>0.99.50 (1.05 Ghz) with ponder, 50 games.
>>
>>In 2-3 months, after my CCE tourney I can make a test.
>>If you have a better idea please write.
>>
>>I think 128 MB for hashtables, and 4pieces tablebases is good enough for the
>>experiment. Tablebases with 4Mb cache. No lean options in configuration files.
>>
>>Best
>>Frank
>
>
>I don't think the experiment is valid...  Because you don't know how much
>crafty and yace are affected by ponder=off.  It is possible that both do a bit
>worse on time management.  So that the matches match pretty well in score.  But
>does that mean ponder=off is then ok?  Only for those two programs.  You have to
>repeat this for _every_ pair of engines you want to test like that...


Until now, I tried to trim the citations in my messages. It takes some editing.
Nobody seems to care. Why?

I think, if everybody would do this, we all could read this forum much more
effectively. If only a minority is doing this, it is just work for this
minority. I give up ...

Then - you have stated many times, that engine matches on one computer make not
much sense. Other people have stated, that it may make sense, and discussed, if
ponder on or off would be more sensible. So, who is right?

You more or less say, that the "one computer may get sensible results" fraction
is wrong. Frank suggests an experiment (on 2 computers). You, from the beginning
call this experiment invalid.

With the same right, the "one computer may get sensible results" fraction can
say, that you should show evidence, that engine matches on one computer, with
ponder off will show no reasonable results. So, they can ask you, to show an
example, where it really makes a difference. How to solve this question?

You allways seem to assume, that all chess engines are only developed with
ponder on in mind. I can say, that this is not true. Yace started with ponder
off, and later I implemented a rather unsophisticated ponder mode. Yace doesn't
have the puzzling mode, that Crafty has. The time management was written, while
Yace had no pondering. I just added very few lines of code (certainly less then
10), to make the engine use more time when pondering is on. So, perhaps, these
two engines would be a good test for the question, how pondering affects engine
strength. Especially, because our approach seems to be very different. I do not
have 2 (reasonable) computers to play engine matches. Neither do I operate an
account at some ICS myself. All my testing is with ponder off.

You, on the other hand, do not care at all about ponder off (which I totally
respect). So, this seems to be a well thought out example match.

Unfortunately, the pages of Volker Pittlik are not available right now. He has
already done many ponder on/off tests. Sure - they wer on one dual computer. My
interpretation of his results were nevertheless, it really does not make a
difference.

And, as a last point. Many people seem to have fun running engine matches. Only
a minor fraction of those will have two computers at hand. Why take them all the
fun?

When sombody sends me Crafty-Yace 1-0 on a single computer and a long time
control, I try to look into the game. It is interesting information for me, as
well as the result of a whole match (I am not too interested in games won by my
program ...).

With best regards,
Dieter Brner






This page took 0.03 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.