Author: Uri Blass
Date: 00:52:51 12/18/02
Go up one level in this thread
On December 18, 2002 at 03:08:03, Bruce Moreland wrote: >On December 17, 2002 at 20:05:48, Omid David Tabibi wrote: > >>Thanks for your comments. We had a very thorough discussion of all the issues >>you've raised, several weeks ago (with interesting comments by Robert Hyatt, >>Gian-Carlo Pascutto, Tony Werten, Uri Blass, etc). I suggest that you first take >>a look at those discussions (check the archives of Nov. 20--30). >> >>Using fixed time instead of fixed depth incurs many problems, e.g., the >>experiment will not be repeatable, and will be heavily hardware dependant, in >>addition to dependance on engine's NPS. Because of all these reasons fixed depth >>experiments are used more frequently for algorithmic comparisons (e.g., see >>Heinz' articles as the most recent examples). > >I got the ICGA today, so this is the first I've heard of this article. > >I don't agree with this means of doing experiments. > >You have version A which gets 10 seconds per position to think on some >hypothetical hardware, and finds 65 solutions. You have version B, which gets >17 seconds on the same hardware, and finds 71 solutions. > >Which is better, A or B? > >You conclude B. > >I disagree with your conclusion, for obvious reasons. *If* Ernst does it the >same way, I disagree with Ernst. And *if* Bob does it that way, I disagree with >Bob. > >Which is my point. If this paper can be juried and still published with this >flaw in it, and if foremost experts in the field can fail here to refute this >methodology, our field is stuck and needs a good kick. > >You cannot conclude that algorithm B is superior to algorithm A, if you give >algorithm B more time to operate. > >The proper conclusion is that you don't know if B is superior to A. > >It is possible that your algorithm is superior. Further testing may show it. >The data as reported do not support your conclusion. > >If you were trying to prove that this is superior to null move R=2, your data >can support that conclusion, since you found more solutions with fewer nodes. >That is a very interesting conclusion. > >But your data do not show that your algorithm is superior to null move R=3, >since you found more solutions with more nodes. An inferior algorithm can do >that. You could have tested the same algorithm twice and proven that it is >superior to itself. Something is wrong here. > >bruce I do not think that the paper claimed to prove that verification search is better than R=3(I agree that it was better to do it but there is a difference between criticizing a paper for not doing it and saying that everything that the author knows is wrong). I believe that it is better than R=3 for a lot of programs(at least at long time control). I think that the interesting question is not if the article is written correctly but if the idea can help programs. If the idea can help programs even when you can criticize the article then saying "everything that you know is wrong" does not seems to me correct. I agree that the fact that articles do not try to compare time is wrong. I understand the problem of the fact that time is not deterministic but it is still possible to use fixed estimated time. The program may have a global varaible with the name estimated_time that may be updated after every procedure so the result can be reproducable and give better estimate for time than the number of nodes. Uri
This page took 0.14 seconds to execute
Last modified: Thu, 07 Jul 11 08:48:38 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.