Computer Chess Club Archives




Subject: Re: Proving something is better

Author: Sune Fischer

Date: 16:16:16 12/18/02

Go up one level in this thread

On December 18, 2002 at 18:23:28, Bruce Moreland wrote:

>On December 18, 2002 at 16:15:52, Sune Fischer wrote:
>>By beef was with those claiming time to solution is all that makes sense.
>>Your original example was to add a bunch of eval and then not count it as nodes,
>>my point was that you then change more than just the search. You alter the whole
>>test bench, there is no base for comparison at all when it's basicly a different
>My example was extreme, and I was hoping to reduce the whole thing to a black
>If all you get to see is what comes out of the box, you draw conclusions about
>what is better or worse based upon what comes out of the box.
>If you have religion about what you *think* is in the box, the conclusions you
>draw are different.
>These algorithms are all processes that take input, consume processor time, and
>produce an output.  You cannot compare their efficiency without factoring in
>differences in processor time.
>If one of them consumes more time, you *expect* it to produce a better result,
>even if it is marginally inferior.  You certainly don't assume it's *superior*.

Well that is where we disagree. I don't believe you can or should see it as a
black box. What works for Genesis might not work for Ferret, you will want
information on why it worked for Genesis. This might make it possible for you to
say right off - this is never going to work for me, because....

Let's say you want to examine generation and search of checking moves in
You run the testset and find that wow, it solved the whole testsuite 15% faster!

Now, what does this tell to me, exactly?
Well, it tells me exactly nothing, because I don't know how much overhead you
spent in generating the checking moves. Did it take you ages to generate these
moves, then maybe I can do it better and get an even larger speedup! Or maybe
your program has structures so this is simple, and I will never get close to
even 1% speedup.

What I really want to know, is how many nodes did it save you?
From that I can guestimate whether it is worth it in my program.

>I was trying to point out that it was easy to redefine a node in order to cause
>one program to take markedly more time.

Okay, let me try again;)

In this case you can't say "program A uses fewer nodes then program B to get to
ply X", because it is _different_ types of nodes! You can't compare them, nodes
in program A is not the same as nodes in program B, it is apples and oranges.
For the comparison to work like in the paper, you need to use the same kind of

No big fuzz here on my account though, I believe this is pretty much equivalent
to time measurements, all the same I prefer nodes (in this case).

>I see this as being approximately as honest as changing the search algorithm so
>the program consumes more time.
>Nodes to solution is fine as long as each node takes the same amount of time,
>otherwise this differs from time to solution, which is what you really want.

This is what you want now, today in your chess program. In 5-10 years things may
have changed and the article is going to be outdated if you aren't careful.

For instance, if I were to write an article about hashing in qsearch, I would
report node counts and not time. Memory latency or hardware architecture could
change in the future and this would certainly have a big effect on the time
measurements, but the nodes are "generic" and will pass the test of time (if
that is even possible in the field of computer science:).


This page took 0.01 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.