Computer Chess Club Archives


Search

Terms

Messages

Subject: Everything you know is wrong

Author: Bruce Moreland

Date: 00:08:03 12/18/02

Go up one level in this thread


On December 17, 2002 at 20:05:48, Omid David Tabibi wrote:

>Thanks for your comments. We had a very thorough discussion of all the issues
>you've raised, several weeks ago (with interesting comments by Robert Hyatt,
>Gian-Carlo Pascutto, Tony Werten, Uri Blass, etc). I suggest that you first take
>a look at those discussions (check the archives of Nov. 20--30).
>
>Using fixed time instead of fixed depth incurs many problems, e.g., the
>experiment will not be repeatable, and will be heavily hardware dependant, in
>addition to dependance on engine's NPS. Because of all these reasons fixed depth
>experiments are used more frequently for algorithmic comparisons (e.g., see
>Heinz' articles as the most recent examples).

I got the ICGA today, so this is the first I've heard of this article.

I don't agree with this means of doing experiments.

You have version A which gets 10 seconds per position to think on some
hypothetical hardware, and finds 65 solutions.  You have version B, which gets
17 seconds on the same hardware, and finds 71 solutions.

Which is better, A or B?

You conclude B.

I disagree with your conclusion, for obvious reasons.  *If* Ernst does it the
same way, I disagree with Ernst.  And *if* Bob does it that way, I disagree with
Bob.

Which is my point.  If this paper can be juried and still published with this
flaw in it, and if foremost experts in the field can fail here to refute this
methodology, our field is stuck and needs a good kick.

You cannot conclude that algorithm B is superior to algorithm A, if you give
algorithm B more time to operate.

The proper conclusion is that you don't know if B is superior to A.

It is possible that your algorithm is superior.  Further testing may show it.
The data as reported do not support your conclusion.

If you were trying to prove that this is superior to null move R=2, your data
can support that conclusion, since you found more solutions with fewer nodes.
That is a very interesting conclusion.

But your data do not show that your algorithm is superior to null move R=3,
since you found more solutions with more nodes.  An inferior algorithm can do
that.  You could have tested the same algorithm twice and proven that it is
superior to itself.  Something is wrong here.

bruce



This page took 0.07 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.