Author: Vincent Diepeveen
Date: 04:47:01 12/18/02
Go up one level in this thread
On December 18, 2002 at 05:49:43, Sune Fischer wrote: >On December 18, 2002 at 03:08:03, Bruce Moreland wrote: > >>You have version A which gets 10 seconds per position to think on some >>hypothetical hardware, and finds 65 solutions. You have version B, which gets >>17 seconds on the same hardware, and finds 71 solutions. >> >>Which is better, A or B? >> >>You conclude B. > >There can be no conclusion, because although version B solves more position it >also uses more nodes. > >>I disagree with your conclusion, for obvious reasons. *If* Ernst does it the >>same way, I disagree with Ernst. And *if* Bob does it that way, I disagree with >>Bob. > >I don't believe he concluded that, I asked Omid that question last time the >discussion was running here, and he said there was no conclusion to be drawn in >that case, which of course is correct. I agree. You can't draw conclusions based upon that. >>Which is my point. If this paper can be juried and still published with this >>flaw in it, and if foremost experts in the field can fail here to refute this >>methodology, our field is stuck and needs a good kick. >> >>You cannot conclude that algorithm B is superior to algorithm A, if you give >>algorithm B more time to operate. >> >>The proper conclusion is that you don't know if B is superior to A. > >The problem as I see it, is that he has split the process in two. Right he has wrongly split it in order to avoid even the worst proofreader from smelling the truth. >First he designs an algorithm to make a smaller tree and then he verifies that >it's also better (solving more positions). In neither of the 2 tests the algorithm is superb. So he could not draw the conclusion that verification search is better. *no way*. >Those are very though demands, you _can_ get an overall improvement by say, >searching 5% more nodes but in return get far less instability in your search. He doesn't hash whether he has done a verification, so his implementation for sure is buggy. I wonder why proofreaders didn't see that. >Such an improvement wouldn't never be discovered by this method, because more >nodes is bad _by definition_. Not always. time to solution counts. Nothing else counts. Of course there is other conditions that must be satisfied too like testing at the same machine. If i do not nullmove in DIEP at 1 ply depthleft, then diep needs less nodes to complete a search, but more time. Number of nodes is trivially bad way to measure things when we talk about nullmove and such methods that all put stuff into the hashtable. Nothing as cheap as a hashtable transposition. Just like 500 clocks or so. There is plenty of other algorithmic changes to figure out which reduce number of nodes and increase time to search. Good other example of what decreased my node count *considerably* but is just too expensive too do is ordering at every position in the main search all moves based upon evaluation of the position after it (of course also using a SEE with it in combination). So ordering it by evaluation. Even if i do not do that at depthleft == 1 ply, then it is slowing me down a factor 2 or so nearly but it reduces node count with around 30% or so? >Like others, I agree that a more "objective" form of measure should be done, ie. >nodes to solution / time to solution. time to get to the same search depth and if you fear you go prune too much then also how many positions you solve within a certain time. Nodes to solution is not acceptible at all. >Personally I prefer nodes to solution, because of the reproducibility. I don't In a parallel search nodes to solution are not reproducable. Also the best parallel algorithm is keeping all but 1 cpu busy (for example dead locked) and letting 1 processor search. A 100% speedup i get then if i just look to nodes to solution. don't you agree that this leads to conclude that we have created the ideal algorithm then? >like having to do averages when it can be avoided. Also if Omid is running on >some grand network machine with different loads from minute to minute, it is >going to be meaningless. that is not the case here clearly. a scientist has to take care his tests are done accurate. So you publish both. simply the entire output of the program. If the entire output of genesis would be on his homepage, we still can draw a diagram whether R=3 solved within the same time more solutions than R=2. So publishing the logfiles of ones program is something you need to do to IMHO. I sure will. >However if he plans to do parallel algorithms, then I think he really neesd a >constant dedicated resource and time to solution measurements. You cannot compare testing at a home computer with parallel testing at a supercomputer. Like Bob explained already it is very hard to get system time. For me it's easier now to perform tests, but i also do not know at which partition DIEP gets scheduled. there is 7 partitions. I am at the mercy of the OS where the program is scheduled. If i ask for 16 processor test of DIEP then it gets scheduled where the resource is free earliest. If that's at the P2 partition which has 32 processors then communication there is a lot faster than at the P7 partitions which has 512 processors. Also it is never the case that the P7 partition is idle. This machine is too expensive to get exclusively used by me for just a few tests. No government can afford that, for sure not the Dutch Government. If you want to know what a test costs that's not so hard. For example for a short 16 processor test of less than 3 minutes that was in total 1800 seconds cpu time used actually. So that was calculated at 0.5 cpu hour. If i do a test at 256 processors for 1 day. that's 256 processors x 24 hours a day x 30 euro a processor hour = 184320 euro. DIEP at world championship 2003: 7 days (they keep entire partition free for me) times 24 hours 512 processors = 86016 processor hour at 30 euro a processor hour = 2580480 Slowly realize the problem? Someone shouting that it ain't 30 euro a processor hour? Ok then do next math: divide half a billion dollar (cost of machine) by the number of processor hours available within a year or 3. Don't forget to also include power, support from SGI, and many salaries of doctors running the thing and expenses of professors and clever scientists taking decisions what happens with the machine. Let's do it for 3 years: 0.5 billion euro / 1024 processors x 350 days (15 days maintenance) x 24 hours times 50% effective usage of the machine at any moment of the day = 500 000 000 / 50%.1024.350.24.3 = 38.75 euro for each processor hour. These numbers are fictive and are not true for the TERAS machine. I invented them. Be clear. It is just an example to give insight in the costs of those machines for each cpu hour. It is clearly you can't demand from supercomputer runs that you always have the same bandwidth and latency for every test. Despite that a researcher should of course try to do his best. Important is to not invent numbers yourself. Or to lose your precious runs. Note that outputs from any run i ever did are saved in special logfiles and error files. No i am not deleting them. Instead i keep track of them and when DIEP's working that well that i feel i must do a few big tests (instead of the 3 minute tests i keep now at at most 16 cpu's) then all those outputs will be freely available for download, upload and spreading around the world as long as they are unmodified. I wonder always where the outputs of the previous parallel researchers are... ...i know they do not get thrown away. The back up facilities at such supercomputers are great! Best regards, Vincent >>It is possible that your algorithm is superior. Further testing may show it. >>The data as reported do not support your conclusion. >> >>If you were trying to prove that this is superior to null move R=2, your data >>can support that conclusion, since you found more solutions with fewer nodes. >>That is a very interesting conclusion. >> >>But your data do not show that your algorithm is superior to null move R=3, >>since you found more solutions with more nodes. An inferior algorithm can do >>that. You could have tested the same algorithm twice and proven that it is >>superior to itself. Something is wrong here. >> >>bruce
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.