# Computer Chess Club Archives

## Messages

### Subject: Re: Proving something is better

Author: Sune Fischer

Date: 13:15:52 12/18/02

Go up one level in this thread

```On December 18, 2002 at 15:32:55, Bruce Moreland wrote:

>I proposed a silly example, and you've gotten stuck on that.  I'll propose a
>more sensible one and see where that gets us.
>
>Take a program, call it A.  A uses R=2, and finds 50 solutions through ply 9.
>
>Take another program, call it B.  B does not use null move at all, and it finds
>55 solutions through ply 9.
>
>Is B better than A?
>You won't argue that B is better than A, because you'll tell me that of course B
>takes much longer to produce its results.

Right!

That is the problem when you try and optimize in a 2-D parameter space.
One parameter improves (more solutions) the other parameter worsens (more
nodes), then there is no sensible conclusion to be drawn.

>But what you are telling me really is that you have some notions, and you are
>willing to let these notions bridge a few gaps:  You believe that if they were
>given equal time, that R=2 would find more than the version with no null move.
>
>But my example doesn't say that.  For all you know, if you give them equal time,
>the R=2 version will find 52 solutions.  The assumption that it will find (well)
>in excess of 55 solutions is a leap of faith.

Then you have not understood what I've been saying.
In your example you can't conclude anything, there is no linear relation between
nodes solved and computation time, necessarily.

>This is what is happening in Omid's paper.

Quite, to some degree.
By beef was with those claiming time to solution is all that makes sense.
Your original example was to add a bunch of eval and then not count it as nodes,
my point was that you then change more than just the search. You alter the whole
test bench, there is no base for comparison at all when it's basicly a different
program.

>R=3 finds 65 solutions in one unit of time.  R=2 finds 66 solutions in 2.2 units
>of time.  Does this support the conventional logic that R=2 is better than R=3?
>That would be a very poor leap of faith.

No that is not conventional logic. As I said, two parameters going off in
opposite directions and you are lost. You need both to point in the same
direction to make a safe conclusion.
This is why I suggested using a single parameter (nodes to solution), so that
you could always conclude something from the experiment. But even that has
certain problems.
Actually I can't think of anything that would be perfect, in all aspects.

>VR=3 finds 71 solutions 1.6 units of time.  VR=3 is shown to be better than R=2,
>but is it shown to be better than R=3?  This is a better leap of faith, but it
>is still a leap of faith.

Yep, but you can go all the way, arguing that tactical testsets are no proof at
all that the algorithm is any good.
However Omid argues that VR is improving stability by design, and it is faster
than R=2. Score 2-0 for VR=3 over R=2.

Whether the nodes to ply and R=3 comparisons are valid I don't know, I would
have to re-read the article. I use R=2/3 so I definitely would need to do the
experiment anyhow.

-S.

>bruce

```