Author: Dave Gomboc
Date: 00:53:48 12/18/02
Go up one level in this thread
On December 18, 2002 at 03:08:03, Bruce Moreland wrote: >On December 17, 2002 at 20:05:48, Omid David Tabibi wrote: > >>Thanks for your comments. We had a very thorough discussion of all the issues >>you've raised, several weeks ago (with interesting comments by Robert Hyatt, >>Gian-Carlo Pascutto, Tony Werten, Uri Blass, etc). I suggest that you first take >>a look at those discussions (check the archives of Nov. 20--30). >> >>Using fixed time instead of fixed depth incurs many problems, e.g., the >>experiment will not be repeatable, and will be heavily hardware dependant, in >>addition to dependance on engine's NPS. Because of all these reasons fixed depth >>experiments are used more frequently for algorithmic comparisons (e.g., see >>Heinz' articles as the most recent examples). > >I got the ICGA today, so this is the first I've heard of this article. > >I don't agree with this means of doing experiments. > >You have version A which gets 10 seconds per position to think on some >hypothetical hardware, and finds 65 solutions. You have version B, which gets >17 seconds on the same hardware, and finds 71 solutions. > >Which is better, A or B? > >You conclude B. > >I disagree with your conclusion, for obvious reasons. *If* Ernst does it the >same way, I disagree with Ernst. And *if* Bob does it that way, I disagree with >Bob. > >Which is my point. If this paper can be juried and still published with this >flaw in it, and if foremost experts in the field can fail here to refute this >methodology, our field is stuck and needs a good kick. > >You cannot conclude that algorithm B is superior to algorithm A, if you give >algorithm B more time to operate. > >The proper conclusion is that you don't know if B is superior to A. > >It is possible that your algorithm is superior. Further testing may show it. >The data as reported do not support your conclusion. > >If you were trying to prove that this is superior to null move R=2, your data >can support that conclusion, since you found more solutions with fewer nodes. >That is a very interesting conclusion. > >But your data do not show that your algorithm is superior to null move R=3, >since you found more solutions with more nodes. An inferior algorithm can do >that. You could have tested the same algorithm twice and proven that it is >superior to itself. Something is wrong here. > >bruce My copy of the issue is at home, but my recollection is that he claimed superiority over R=2, but did not claim superiority over R=3. (Or if he did, then this was based on experiments by others [notably Heinz] who already demonstrated the superiority of R=2 over R=3. I agree that the data presented does not justify that claim without additional information.) Dave
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.