Author: Sune Fischer
Date: 02:49:43 12/18/02
Go up one level in this thread
On December 18, 2002 at 03:08:03, Bruce Moreland wrote: >You have version A which gets 10 seconds per position to think on some >hypothetical hardware, and finds 65 solutions. You have version B, which gets >17 seconds on the same hardware, and finds 71 solutions. > >Which is better, A or B? > >You conclude B. There can be no conclusion, because although version B solves more position it also uses more nodes. >I disagree with your conclusion, for obvious reasons. *If* Ernst does it the >same way, I disagree with Ernst. And *if* Bob does it that way, I disagree with >Bob. I don't believe he concluded that, I asked Omid that question last time the discussion was running here, and he said there was no conclusion to be drawn in that case, which of course is correct. >Which is my point. If this paper can be juried and still published with this >flaw in it, and if foremost experts in the field can fail here to refute this >methodology, our field is stuck and needs a good kick. > >You cannot conclude that algorithm B is superior to algorithm A, if you give >algorithm B more time to operate. > >The proper conclusion is that you don't know if B is superior to A. The problem as I see it, is that he has split the process in two. First he designs an algorithm to make a smaller tree and then he verifies that it's also better (solving more positions). Those are very though demands, you _can_ get an overall improvement by say, searching 5% more nodes but in return get far less instability in your search. Such an improvement wouldn't never be discovered by this method, because more nodes is bad _by definition_. Like others, I agree that a more "objective" form of measure should be done, ie. nodes to solution / time to solution. Personally I prefer nodes to solution, because of the reproducibility. I don't like having to do averages when it can be avoided. Also if Omid is running on some grand network machine with different loads from minute to minute, it is going to be meaningless. However if he plans to do parallel algorithms, then I think he really neesd a constant dedicated resource and time to solution measurements. -S. >It is possible that your algorithm is superior. Further testing may show it. >The data as reported do not support your conclusion. > >If you were trying to prove that this is superior to null move R=2, your data >can support that conclusion, since you found more solutions with fewer nodes. >That is a very interesting conclusion. > >But your data do not show that your algorithm is superior to null move R=3, >since you found more solutions with more nodes. An inferior algorithm can do >that. You could have tested the same algorithm twice and proven that it is >superior to itself. Something is wrong here. > >bruce
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.