Author: martin fierz
Date: 17:06:36 09/05/02
Go up one level in this thread
On September 05, 2002 at 19:08:54, Matthew Hull wrote: >On September 05, 2002 at 18:09:44, martin fierz wrote: > >>On September 05, 2002 at 17:09:44, Matthew Hull wrote: >> >>>On September 05, 2002 at 16:43:16, martin fierz wrote: >>> >>>>On September 05, 2002 at 13:43:10, Matthew Hull wrote: >>>> >>>>>On September 05, 2002 at 13:28:20, Miguel A. Ballicora wrote: >>>>> >>>>>>On September 05, 2002 at 10:05:05, Robert Hyatt wrote: >>>>>> >>>>>>>On September 05, 2002 at 00:25:58, Miguel A. Ballicora wrote: >>>>>>> >>>>>>>>On September 04, 2002 at 18:38:17, Dann Corbit wrote: >>>>>>>> >>>>>>>>>My take on the matter (in one paragraph): >>>>>>>>>Robert wrote a paper on parallel speedup, showing a 1.7 increase for 2 CPU's (as >>>>>>>>>derived from his more general formula). Vincent was unable to reproduce this >>>>>>>>>sort of speedup, and thought the research was faulty. Robert agreed that the >>>>>>>>>test set was limited and you won't always get that sort of speedup, but as an >>>>>>>>>average (over a broad set of positions) that's about what he got. There has >>>>>>>>>been some acrimony over whether superlinear speedups are possible. I think that >>>>>>>>>the jury is still out on that one. >>>>>>>>> >>>>>>>>>At any rate, that's my take on the whole thing. >>>>>>>>> >>>>>>>>>Vincent always sees things in pure, jet black or gleaming, powder white. If >>>>>>>>>something isn't terrific, then it is pure junk. While I think his mode of >>>>>>>>>interesting is a bit odd, it's one of the things that make Vincent interesting. >>>>>>>> >>>>>>>>He crossed the line when he used the word "fraud" and "lie" >>>>>>>>to describe a scientific paper without any solid proof (he only proved a flaw in >>>>>>>>the presentation). Too serious. >>>>>>>> >>>>>>>>To be honest, I am embarrassed to be reading this thread. One side does not >>>>>>>>recognize a flaw (it could be honest and I believe it, happens many times, big >>>>>>>>deal) and the other makes pathetic accusations of fraud mixing it up with old >>>>>>>>issues (Deep blue etc.). To top it all, ad hominem attacks. >>>>>>>> >>>>>>>>In this conditions it is impossible to discuss anything. >>>>>>> >>>>>>>While I understand what you mean, I don't see any major "flaw". >>>>>> >>>>>>No, it is not major, it is a very minor flaw in the presentation. Not a big >>>>>>deal, but you cannot stand there and say that it is just ok. You cannot say that >>>>>>it is ok the way you rounded it and everything is justified by the big >>>>>>variability. The only thing that the big variability shows is that the flaw is >>>>>>minor, but it does not show that there is no flaw in the presentation. In those >>>>>>cases standard deviations should be shown using numbers that were rounded >>>>>>properly. >>>>>> >>>>>>Don't get me wrong, I understand and accept completely everything you say, if I >>>>>>were accused of fraud I would go overboard myself. But please, do not try to >>>>>>convince us that those tables are the proper way to present something. >>>>> >>>>>Under the circumstances, I don't think he had a choice. It was the only way to >>>>>add the data so long after the fact at the request of the referees. Was he >>>>>supposed to say in the paper that the refs wanted the data so I was forced to >>>>>extrapolate it? What would you have done in that situation? >>>> >>>>of course he had a choice! when a referee asks you for something more than what >>>>you have in your paper, you can either comply, and produce that data, or you can >>>>try to convince him that it's not important. even if you fail to convince the >>>>referee, the editor of the journal has the last word about publication, and if >>>>your paper is really good (it is), then you can also survive a hostile referee. >>>>bob should either have rerun his tests and saved his logs to produce real data, >>>>or he should have convinced the referee that this data is unnecessary (it is!), >>>>and if that failed, he should have admitted to the journal editor that he did >>>>not have the raw data any more, and asked him to publish it all the same. >>>>making up data (to please a referee or for other reasons) and passing it on as >>>>actual data is probably the biggest no-no of all in science. >>> >>>"Making up data" is too strong here. The times and nodes are functions of the >>>speedup. Deriving them from this function is not "making up" data, it's >>>calculating the data from a known function. >>> >>>Is this not standard practice, especially under the circumstances described >>>above? >> >>making up data is not too strong. of course the numbers in the table are very >>close to what bob actually measured. but the table says: "this is the time in >>seconds that the program needed on this position" - and the numbers in the table >>are not what it says. even if it is very close to the truth, it is still made >>up. this is definitely NOT standard practice - under no circumstances do you >>EVER make up data. even if it is very close to the true data.... > >To say that the times were "made up" is to say that the speedups were "made up", >since the times were derived directly from the speedups. > >You are saying the speedups are "made up". you don't understand... read carefully: i never said that anything bob did invalidates any conclusion he made in the article. i said the opposite. i am definitely not saying the speedups are made up. vincent says that. i don't. if you read my posts, you will see that i never say the speedups are made up... the point is that the data in the table is not real, measured data. bob had real data, calculated the speedups, rounded them, and later recalculated the times in that table with the rounded numbers (if the speedup was not rounded, you would be right, then his numbers would be real - but they're not). yes, the numbers in this table are off only by a few % from the real numbers, but that doesn't change the fact that they are not what he claims to be. if the table says "i measured these numbers", and it turns out that they are not measured numbers, then it is wrong. in science, you just don't do that... for the reasons i gave to you. OF COURSE this is only a very small fudging of reality. but if you are willing to fudge very little, you might also be willing to fudge a little bit more. and where do you draw the line? obviously, if you are not in science, this may all be a bit too much nitpicking for you. big business doesn't work this way (enron, worldcom, you name it). science does. aloha martin PS: >Making up is pure invention "ex nihilo". Don't exaggerate. that is your definition of "making up". i say: putting any number there that you did not actually measure (even if it is close to the true number) is "making up". you write something down which you didn't measure. i don't care how close to the true number it is. again, that is the way it works in science. if you want to publish such results on a webpage, fine. if you publish that kind of stuff in a scientific journal, not fine. >> >>let me give you an example: i'll claim i made an experiment measuring the time >>it takes for a stone to drop from x meters height. thanks to physics, i already >>know how long it will take. so i give you a table, with numbers in, which, if >>you actually repeat the experiment, will turn out to be true. am i a fraud if i >>do this? certainly. i claim to have measured something, which i haven't in fact. >>it doesn't matter whether my numbers are correct or not, it only matters that i >>pretend to have measured something i didn't really measure. >> >>uri puts it very well: science is about trusting another person's results. if i >>see that bob publishes data which he didn't really measure, i lose some of my >>faith in his results. so you say: "well, this was just a minor flaw which was >>none of his fault, because a referee made him do it." the problem is this: bob >>was willing to cheat a little bit just to satisfy the referee. so how do you >>know that the next time a referee asks him to add something he will not cheat a >>little bit more? and if he gets away with that too? what will he do next? >>i'm not suggesting that he did anything else wrong, or that he will do anything >>else wrong in the future (in fact, i trust bob's results very much, and will >>continue to do so), i just want to explain to you why it is important in science >>that you never publish anything you didnt really measure. >> >>aloha >> martin >> >>> >>>>people routinely >>>>get fired for doing that in academia. even if it is totally irrelevant data, as >>>>in this case, you just don't do that. >>>> >>>>it is no coincidence that miguel and i who say bob did something wrong are both >>>>in research... >>>> >>>>bob's argument that it's irrelevant, which convinces both of us, should have >>>>been used to counter the referee's request. >>>> >>>>aloha >>>> martin >>>> >>>> >>>> >>>>>>What I accept is that it does not change a bit the relevance of the ideas. >>>>>> >>>>>>Regards, >>>>>>Miguel >>>>>> >>>>>> >>>>>>>As in the >>>>>>>question I posed to vincent, if I ask you to measure the radius, diameter >>>>>>>and circumference of a piece of pipe, once you have one, you can derive the >>>>>>>others, but there will be a tiny accuracy issue. Because pi factors into >>>>>>>the relationship among the three values, and what value do you use for pi? >>>>>>>So even in something so simple as diameter vs circumference, there are a >>>>>>>_set_ of possible diameters that will yield the same circumference, to some >>>>>>>particular number of decimel places of accuracy. Yet in reality, only _one_ >>>>>>>of those diameters can be the _real_ diameter. >>>>>>> >>>>>>>After showing Martin just how "bad" smp variability can be, that equates to >>>>>>>a piece of pipe made out of some _very_ temp-sensitive material, so that the >>>>>>>very act of touching it to measure it causes a change in the >>>>>>>diameter/circumference. And that is what I see in SMP results _all_ the time. >>>>>>> >>>>>>>IE for a given actual time for a 16 processor test, and a computed speedup >>>>>>>made by dividing the raw 1-cpu time by the raw 16-cpu time, you can now go back >>>>>>>and re-compute the 1-cpu raw time. And there will be a _set_ of answers to >>>>>>>that computation that still produce the same 16-cpu speedup. Is one of those >>>>>>>pre-computed 1-cpu times (which can vary in the low order digits when the >>>>>>>value is in the thousands) any better than another? Is one better because you >>>>>>>computed it, and you actually observed it in the game under question? What if >>>>>>>you run it again and you get _another_ one of the computed 1-cpu times, as it >>>>>>>is easy to see the one-cpu time vary by a second or so over a long period, >>>>>>>due to network traffic, interrupts, and so forth, maybe even a daemon process >>>>>>>waking up for a moment to do something? >>>>>>> >>>>>>>My point? >>>>>>> >>>>>>>If you take a 16-cpu speedup, and a 16-cpu time, and use that to derive >>>>>>>the set of 1-cpu times (measured only in seconds) you will get a set of >>>>>>>N times that could produce that same speedup given the observed 16-cpu time >>>>>>>and computed 16-cpu speedup. The inverse is the better way of thinking about >>>>>>>this of course, because that 16 cpu time is just a "snapshot" that could vary >>>>>>>dynamically, and as the 16-cpu time varies, so would the set of 1-cpu times >>>>>>>that would still produce that same speedup. >>>>>>> >>>>>>>In other words, there is a _lot_ of inaccuracy already. So the question >>>>>>>becomes, "why compute some of the times, why not observe them all?" That is >>>>>>>"the" question here. And it is one I am not sure I can answer yet. I have >>>>>>>gotten "closer" to the disk failure date. From info in my files at the office. >>>>>>> >>>>>>>I bought a Gateway G6/200 (pentium pro 200) when they first came out. I think >>>>>>>Bruce got his a month or so before me. In early 1996, I had a disk crash, >>>>>>>and called gateway for a replacement (this is the part I have docs for as you >>>>>>>will see shortly). They sent one but when it arrived, it was a 68-pin wide >>>>>>>scsi drive, while the machine they had sold me 2 months earlier was a 50-pin >>>>>>>narrow scsi drive (this machine had 4 4.5 gig scsi drives). I called them to >>>>>>>tell them about the "error" (I couldn't use that drive with my scsi controller) >>>>>>>and they responded "we no longer have any narrow scsi stuff, we have migrated >>>>>>>to wide scsi." I responded "great, so I have a machine 2 months old, that >>>>>>>cost me $5,000, and you can't get me a replacement drive? Mark me off the >>>>>>>gateway customer list (I used to order 20-30 machines _per year_ for our >>>>>>>department labs)." That got their attention, they sent me three more wide >>>>>>>scsi drives and a new controller. So I have at least homed in on when I lost >>>>>>>"the world". And that suggests that we _had_ to reconstruct the data as best >>>>>>>we could, although neither of us can say "yes that was the reason" with 100% >>>>>>>reliability. I mainly remember trying to get Crafty back up to speed. I was >>>>>>>getting ready for the WMCCC event, and was not releasing new versions. As a >>>>>>>result, when the disk went down, I lost everything, and had to back up to the >>>>>>>last released version and start over, trying to remember the changes I had >>>>>>>made. >>>>>>> >>>>>>>I don't know when I started back to work on the dts paper, because the panic >>>>>>>about the WMCCC and losing things is the thing that I _really_ recall (not >>>>>>>to mention class assignments, handouts, etc.) But it is certainly within >>>>>>>reason to think that led to the time computations. As I have said all along, >>>>>>>the node computation was unavoidable from the start... >>>>>>> >>>>>>>So, as far as I am concerned, there is _nothing_ "false" in the DTS paper. I >>>>>>>computed the speedups in the normal way, using raw data, back in early 1994 >>>>>>>after the tests were run. That was the "critical data" presented in the paper. >>>>>>>Later, we extrapolated the nodes when they were asked for, and it would seem >>>>>>>that we extrapolated the times at the same time. >>>>>>> >>>>>>>That is about all I can remember, so far. I don't consider the "node" data to >>>>>>>be very interesting at all, and the "time" data is dynamic enough due to the >>>>>>>variability in speedup that any value I published, whether it be extrapolated or >>>>>>>observerd, could likely _never_ be repeated again. As a new test run would >>>>>>>certainly vary by seconds if not much more (another thread in this discussion >>>>>>>shows some wild behavior on kopec 2, and some rock-solid behavior on kopec 3, >>>>>>>to illustrate this point.) >>>>>>> >>>>>>>I don't see it as being a "fraud" at all, but then I'm not Vincent. I think >>>>>>>the concept of "if you can't make yours look better, then make someone else's >>>>>>>look worse" is extremely sad, of course. But he will continue to do that on >>>>>>>all fronts. "crafty's eval is simple". "fritz knows shit about chess". >>>>>>>"tiger is stupid in endgames." We've all seen those comments over and over. >>>>>>>As well as "diep has the most sophisticated eval of any chess program" or >>>>>>>"diep is the strongest program in the world at correspondence time controls". >>>>>>> >>>>>>>What more could anyone say? >>>>>>> >>>>>>>Will Crafty ever approach CB's DTS speedup? I doubt it, because I am not >>>>>>>willing to give up the recursive search, which would be the first thing to >>>>>>>go. If I had high expectations of running on a 32 cpu machine all the >>>>>>>time, then I certainly might consider it. But I suspect that I will be >>>>>>>using a quad for the foreseable future, with maybe an 8-way or 16-way >>>>>>>box on rare tournament occasions. I'm old enough to try to avoid >>>>>>>headaches, which you get with non-recursion when you try to read it later. >>>>>>> >>>>>>>Will someone else beat the DTS result? Most certainly. Even as I wrote the >>>>>>>paper I noticed a couple of things that ought to be changed. Will it bother >>>>>>>me when they do? Not at all. My EPVS algorithm beat the old PVS by a small >>>>>>>amount. Schaeffer tried a new approach that was better than EPVS. Which led >>>>>>>me to try something even better since I had a lot of time that I would be >>>>>>>spending on my dissertation anyway, so that seemed like a natural project to >>>>>>>undertake. Will I do that again? I don't think so. Too much development >>>>>>>time. multiple years of debugging, including one full-time year as I finished >>>>>>>the dissertation work and started the writing. The current SMP approach in >>>>>>>Crafty is not half-bad. And it was not hard to write at all. Those two points >>>>>>>say a lot... >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>Regards, >>>>>>>>Miguel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>Robert has always been a man of strong convictions, and if you call him a >>>>>>>>>'noo-noo head' he'll call you one back. He isn't one to back down when he >>>>>>>>>thinks he is right. That's one of the things I like about Dr. Hyatt. >>>>>>>>> >>>>>>>>>When these two styles happen to ram into one another, the sparks are sure. A >>>>>>>>>philosophical question is often asked: >>>>>>>>>"What happens when an immovable object meets an irresistable force?" >>>>>>>>> >>>>>>>>>The 'debate' is an answer to that question. >>>>>>>>>;-)
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.