# Computer Chess Club Archives

## Messages

### Subject: Re: Proving something is better

Author: Robert Hyatt

Date: 08:19:08 12/19/02

Go up one level in this thread

```On December 18, 2002 at 23:24:08, Bruce Moreland wrote:

>On December 18, 2002 at 22:43:45, Robert Hyatt wrote:
>
>>On December 18, 2002 at 17:18:56, Bruce Moreland wrote:
>
>>>But this doesn't address the problem I pointed out with Omid's thing:
>>>
>>>A takes 30 seconds, produces 50 answers.
>>>
>>>B takes 40 seconds, produces 55 answers.
>>>
>>>It makes absolutely no sense to say that B is better than A, and if the time
>>>differential is large enough, and the difference in number of solutions is small
>>>enough, it may make sense to say the reverse.
>>
>>
>>I don't disagree there...
>
>I've lost Omid, so I'll try to get you to see what I'm talking about then.
>
>Omid proves that with his program, VR=3 is better than R=2.
>
>He assumed before he started that R=2 is better than R=3.  So he never tested
>R=3 to create a baseline, before tweaking R=3 to create VR=3.

I thought he left that question "open" in his paper, because as you said,
he didn't test it.  In particular, "adaptive null-move (variable R)" was
omitted.

As for the R=2 vs R=3 question, that seems to be "open" and wasn't answered,
but one has to ask "does he have to answer _every_ possible question (which
would be desirable but is it a necessary condition for valid testing?) or can
he simply test against a known algorithm that many of us used for a long time,
namely R=2?

Not testing with adaptive R=2/3 is an omission.  Is it a _fatal_ omission?

>
>But he does include numbers for R=3 for a couple of test suites, and the
>solution numbers are almost identical with R=2.  Of course R=3 takes about 40%
>as much time as R=2, so these numbers are superior.
>
>So for these suites, R=3 is better than R=2 for this program.  How do you
>compare a variant of R=3 against anything when your program is already acting as
>if R=3 is better than R=2?  The numbers are either bogus or point to a larger
>conclusion.
>
>bruce

I wouldn't argue against your conclusion except for the "numbers are bogus".
And perhaps he should have done more work with a real R=3 and R=2/3 type
program.  However, in a narrow context, namely the idea of the "verification
search" he did do pretty well in showing that it was no worse than R=2 in terms
of nodes searched, which is the way I looked at this...

```