Computer Chess Club Archives




Subject: Re: Everything you know is wrong

Author: Dave Gomboc

Date: 00:53:48 12/18/02

Go up one level in this thread

On December 18, 2002 at 03:08:03, Bruce Moreland wrote:

>On December 17, 2002 at 20:05:48, Omid David Tabibi wrote:
>>Thanks for your comments. We had a very thorough discussion of all the issues
>>you've raised, several weeks ago (with interesting comments by Robert Hyatt,
>>Gian-Carlo Pascutto, Tony Werten, Uri Blass, etc). I suggest that you first take
>>a look at those discussions (check the archives of Nov. 20--30).
>>Using fixed time instead of fixed depth incurs many problems, e.g., the
>>experiment will not be repeatable, and will be heavily hardware dependant, in
>>addition to dependance on engine's NPS. Because of all these reasons fixed depth
>>experiments are used more frequently for algorithmic comparisons (e.g., see
>>Heinz' articles as the most recent examples).
>I got the ICGA today, so this is the first I've heard of this article.
>I don't agree with this means of doing experiments.
>You have version A which gets 10 seconds per position to think on some
>hypothetical hardware, and finds 65 solutions.  You have version B, which gets
>17 seconds on the same hardware, and finds 71 solutions.
>Which is better, A or B?
>You conclude B.
>I disagree with your conclusion, for obvious reasons.  *If* Ernst does it the
>same way, I disagree with Ernst.  And *if* Bob does it that way, I disagree with
>Which is my point.  If this paper can be juried and still published with this
>flaw in it, and if foremost experts in the field can fail here to refute this
>methodology, our field is stuck and needs a good kick.
>You cannot conclude that algorithm B is superior to algorithm A, if you give
>algorithm B more time to operate.
>The proper conclusion is that you don't know if B is superior to A.
>It is possible that your algorithm is superior.  Further testing may show it.
>The data as reported do not support your conclusion.
>If you were trying to prove that this is superior to null move R=2, your data
>can support that conclusion, since you found more solutions with fewer nodes.
>That is a very interesting conclusion.
>But your data do not show that your algorithm is superior to null move R=3,
>since you found more solutions with more nodes.  An inferior algorithm can do
>that.  You could have tested the same algorithm twice and proven that it is
>superior to itself.  Something is wrong here.

My copy of the issue is at home, but my recollection is that he claimed
superiority over R=2, but did not claim superiority over R=3.  (Or if he did,
then this was based on experiments by others [notably Heinz] who already
demonstrated the superiority of R=2 over R=3.  I agree that the data presented
does not justify that claim without additional information.)


This page took 0.06 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.