Author: Robert Hyatt
Date: 12:43:01 09/05/02
Go up one level in this thread
On September 04, 2002 at 15:47:21, Uri Blass wrote: >On September 04, 2002 at 15:20:55, Rolf Tueschen wrote: > >>On September 04, 2002 at 08:47:59, Uri Blass wrote: >> >>>On September 04, 2002 at 08:11:28, Rolf Tueschen wrote: >>> >>>>On September 04, 2002 at 07:57:35, Uri Blass wrote: >>>> >>>>>On September 04, 2002 at 07:15:42, Ralf Elvsén wrote: >>>>> >>>>>>On September 03, 2002 at 18:29:33, Gian-Carlo Pascutto wrote: >>>>>> >>>>>>>On September 03, 2002 at 18:15:50, Robert Hyatt wrote: >>>>>>> >>>>>>>>On September 03, 2002 at 18:03:14, Gian-Carlo Pascutto wrote: >>>>>>>> >>>>>>>>>However reasonable your explanations may be, the gist of your DTS article >>>>>>>>>and the most important thing for comparison were the speedup numbers. After >>>>>>>>>what we discovered and what you just posted, it is clear that they are >>>>>>>>>based on very shaky foundations. >>>>>>>> >>>>>>>>How so? The speedup numbers were _directly_ computed by dividing times. >>>>>>> >>>>>>>But what times? Certainly not the times you reported. All we have is 5 >>>>>>>speedup numbers that nobody every reproduced and likely will never >>>>>>>reproduce due to the hardware involved. >>>>>>> >>>>>>[SNIP] >>>>>>> >>>>>>>I don't know if you faked the results to look better or not. Maybe I don't >>>>>>>want to know. But whatever be of it, there is little scientific ground >>>>>>>to keep them standing, IMHO. >>>>>>> >>>>>>>-- >>>>>>>GCP >>>>>> >>>>>>The times should have been included, since they are the raw data. Moreover so, >>>>>>since some "unconventional" rounding seem to have been done when computing >>>>>>the speedups. >>>>>> >>>>>>However, regarding "reproducible" and "faked" , how would the inclusion of >>>>>>the times make you happier? They wouldn't be more or less reproducible than >>>>>>the speedups, and they could very well be faked (which I don't believe). >>>>>> >>>>>>Please don't mix the issues of poor data presentation with the scientific >>>>>>intergrity of the work. >>>>>> >>>>>>Ralf >>>>> >>>>>Of course it was possible to fake also >>>>>times without mistakes that people >>>>>are going to find. >>>>> >>>>>The point is that we usually trust >>>>>scientists not to do it but we also trust them >>>>>not to report wrong numbers. >>>>> >>>>>After finding that the numbers about time >>>>>were wrong the trust is broken. >>>>> >>>>>I do not say that the speed up were faked but >>>>>only that I do not know what to believe and >>>>>I agree with GCP that there is little scientific >>>>>ground to keep them standing. >>>> >>>>It's sad to read such sloppy remarks, Uri. >>>>You don't know what to believe, but you agree >>>>that there is little scientific ground... >>>>Is this sound reasoning for you? >>> >>>Yes >>> >>>Science is about proved things. >>>If the experiment cannot be reproduced then >>>we need to decide if to trust the authors. >> >>Yes and no, Uri! >> >>Science is about proving things. Or do you insist on proved things. Because then >>this is false. Science cannot _prove_ things. If we take it literally. The point >>is that any time someone could find something new or a refutation or new >>paradigms. >> >>I don't want to teach you lessons here but believe me the above. Now, you could >>do a lot of things - - but the most important law is that you should >>describe=make documentation of what you did. Now, would you agree that Vincent >>couldn't reproduce Bob simply because of what?? Please try to find the simple >>answer for yourself. But even this isn't the point. Bob said that the DTS >>article just was a very small part of his dissertation. Etc. So, I fear we must >>at first define, what exactly you wanted to be reproduced!? >> >>I could continue like this for several pages. >> >>Let me comment on your conclusion above. You are badly wrong. If we can't >>reproduce we are not yet forced to decide if we coul >>>d trust the former author. Not at all. At first we must analyse _why_ we can't reproduce it. Many aspects. >> >>I will stop it here, because your statement alone showed me that you are miles >>away from the core of the problem. At first you must rethink all the points up >>to here. And then we'll see if you still have a problem. If yes, then we'll >>continue the debate. But the way Vincent believed it should go is surely wrong. >>And as far as I could follow Giancarlo, he was supporting Vincent. > >There is a difference between what GCP said and what Vincent said. > >I understood that GCP's opinion is that we cannot trust the speed up but he did >not say that the speed up was faked. > >Saying that the results have no scientific value is not the same as saying >that Bob lied in order to hide data that he did not like. > >> >>>In most cases I trust articles that >>>were published even without proofs but >>>in this case there are reasons not to do it. >> >>You should never trust, Uri! :) > >I agree that people should try to reproduce the results but in case there is no >way to reproduce the result for clear reasons I prefer to trust the results >unless I can find something wrong in the same article or something wrong with >the past of the author. > >>And honestly, here we are not even close to have put Bob into a real defense. I >>can say that although I am not a programmer or such. Of course I didn't read the >>dissertation, but surely there all the necessary stuff is mentioned. >> >>> >>>> >>>>NB this is not about creating or joining parties, >>>>it's about some basic questions in science and >>>>logic too. At the moment nothing is proven. >>>>Something is looking odd, but Vincent spoke >>>>of lies and fakes and mass fraud. Are you >>>>supporting such verdicts? >>>> >>>>Rolf Tueschen >>> >>>No >>> >>>I supported GCP >>>I did not support Vincent's claims. >> >>I fear that you are on the wrong side this time. As you know I started to claim >>the lack of science etc in the case of DB2 in 1997, but here I don't see why Bob >>should lie or fake or commit mass fraud. What for? > >I do not say that Bob lied but he had an interest to give wrong results to make >cray blitz look better. I don't follow. As of my dissertation date in 1988, nobody had produced a better speedup than I did there. Fortunately, I have all the raw data for that paper. Unfortunately it is a stack of laser printed pages about 6" thick. There was no real reason to "lie" about the results. Jaap asked me to write a paper on DTS after several had asked about it after reading a review of it in the JICCA. I said I would. Then the idea of using a real game came up in conversations at the 1993 event, when I was asked about speedup in games vs speedup in test problem sets. If you choose to not trust the speedups, that is fine by me. I won't lose any sleep over it because I _know_ they are correct. If you are unhappy about the times and nodes, then I can't say more. They are not _the_ observed data, but as I said, most would accept that the diameter of a circle can be produced by dividing the circumference by pi. But you will _never_ produce the same number someone would get with a ruler or micrometer. But is it a _wrong_ number to extrapolate D from C? Not in my book. And as I said, in parallel search, the circle is changing dramatically all the time, so that trying to compute D to 2-3 decimel places just produces random numbers in the low order digits anyway... But enough... Take it or leave it. I've done my part... > >My opinion is that the data that we have is enough to have serious doubt >about the results. Based on what? The results were _the_ important part, and they are certainly correct for the test runs we did. They certainly would likely be a bit different if we could run the test again. That is SMP search. But to say the numbers that were used to produce the nodes/times is not trustworthy because the nodes/times were not observed data doesn't strike me as reasonable. But that is your decision, obviously... > >The problem is that there was wrong or misleading information in the article. >The article did not say that the times were estimated times. It might or might not have said that. Most of the last year was spent rewriting things to make it shorter. I apparently shorted the "test setup" enough that it omitted a few important details. It is possible that was eliminated as well, as trying to explain the node count problem takes a good bit of text and the emphasis was "make it shorter"... I have no original copies to verify what I said about the data, so again, more speculation is all I can offer. But it was written too long ago for me to be sure of any small details... > >This is enough not to trust the results even without assuming that Bob lied >because it may be possible that there is another important data that Bob does >not remember. Such as? there are lots of potential ways to 'attack' the data. The pondering setup. Hash tables. I think my test setup actually _hurt_ parallel performance, for example, because the one cpu test got to ponder 16x longer than the 16cpu test did. But the 1cpu test pondered "better" nodes because the 16cpu test had duplicate work. I chose to accept this because it means that it is possible that the 16cpu speedup could have been a bit better. But it seemed reasonable to give a "lower bound estimate" to be conservative, which I chose to do... Good decision? Hard to say. It could be argued that the extra pondering on the 1cpu (or 2 or 4 or 8) run hurt 1cpu performance in some strange way so that the speedup was overstated. I don't believe it, and I could at least check it with Crafty, but that could be argued and it would be impossible to prove that it wasn't... > >I expected to get only correct information in the article. >After finding that it is not the case I have to agree that we cannot trust the >explantions that came after it. If you notice, my explanation was later matched several times by people posting that "he might have had to reverse-engineer the data for some reason and that would explain this." The conclusion was certainly right about the nodes. And apparently right about the times. Whether you believe that is what happened or not is a personal choice only you can make, however... > >It does not mean that the explanation is wrong and I do not know. > >Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.