Author: Robert Hyatt
Date: 19:41:59 09/03/02
Go up one level in this thread
On September 03, 2002 at 20:38:11, martin fierz wrote: >On September 03, 2002 at 20:14:49, Robert Hyatt wrote: > >>On September 03, 2002 at 18:49:08, martin fierz wrote: >> >>>On September 03, 2002 at 18:26:09, Robert Hyatt wrote: >>> >>>>On September 03, 2002 at 17:44:26, martin fierz wrote: >>>> >>>>>On September 03, 2002 at 16:26:33, Robert Hyatt wrote: >>>>> >>>>>>On September 03, 2002 at 15:50:45, Matthew Hull wrote: >>>>>> >>>>>>>On September 03, 2002 at 15:42:08, Gian-Carlo Pascutto wrote: >>>>>>> >>>>>>>>http://sjeng.sourceforge.net/ftp/hyatt1.png >>>>>>>> >>>>>>>>I will try to upload the full article as soon as I can. >>>>>>>> >>>>>>>>-- >>>>>>>>GCP >>>>>>> >>>>>>>You've got to be kidding. When Vincent posts the numbers, he's got all these >>>>>>>trailing zeros. What's with that? >>>>>>> >>>>>>>It is Vincent that's faking numbers here, not Bob. Bob's numbers are just >>>>>>>rounded off. >>>>>>> >>>>>>>Vincent is the one emitting bogons here. >>>>>> >>>>>> >>>>>>If you didn't see my response elsewhere, this output was produced by a program >>>>>>I wrote to "eat" Cray Blitz log files. I did this for my dissertation as I >>>>>>produced tens of thousands of test position logs for that. >>>>>> >>>>>>I believe that it does something similar to what I do today, which means that >>>>>>anything between 1.71 and 1.8 is treated as 1.8, 1.81 to 1.9 is 1.9. >>>>> >>>>>that sure sounds like a bad thing to do! you should retain 2 digits, since >>>>>the interesting number is not the speedup (1.8, 1.9, whatever), but rather the >>>>>difference to 2. >>>> >>>>I don't use a "log eater" today. I generally run a set of positions, and >>>>get a time for one processor, then a time for 2 processors and divide the two >>>>by hand. I do round to one dec. place as the numbers are already very >>>>unstable, and going to more accuracy is really pointless... >>> >>>well, that is not the point! in the end, you give an average of the speedup >>>which turns out to be 2.0. now, even if your single numbers are unstable, the >>>average of them is much more stable, and could do with more than 1 digit. >>> >>>if you do your wrong rounding twice, first for the single measurement, and then >>>for the final result (you do, as i can see from the page GCP posted), then you >>>can actually make a larger mistake. take measurements 1.81, 1.81, 1.81 1.81 and >>>1.91. you round to 1.9 1.9 1.9 1.9 and 2.0. for the final result you round again >>>to 2.0. when the average of this thing was 1.83. of course, i have chosen my >>>numbers carefully, but there is simply no reason to do the rounding the way you >>>do. on average, with your double rounding, you are giving yourself a 0.1 speedup >>>which is not there! if you thought it was a good idea, and that it did not >>>affect the result really, you should have done it the other way round - down >>>instead of up. then at least you would be conservative... >>> >>>>> so if you measure 1.91 or 1.99, you really measure 0.09 or >>>>>0.01, and obviously these two numbers are quite different. >>>>>also, if you really feel like rounding, you should do it in the normal sense, >>>>>like 1.750 - 1.849 -> 1.8. i don't believe a word of vincent's post, but always >>>>>rounding upwards is definitely making your data look better than it is, and >>>>>should be rejected by any reviewer or thesis advisor - if he knew you were doing >>>>>that :-). not that it makes a big difference of course - but for the 2-processor >>>>>case we are talking about a potential >10% error, which where it starts getting >>>>>significant. >>>>> >>>>>aloha >>>>> martin >>>>> >>>> >>>>or really 5% max? ie 1.9 to 1.99 is only 5% difference... >>> >>>well, that's your interpretation: you say your are measuring x=1.9 or x=1.99. >>>but your experiment is designed to produce numbers between 1 and 2. so if you >>>look at it that way, your result is 1+x, delta_x being the .9 to .99 difference >>>which is 10%. this also makes sense from a practical point of view, since the x >>>thing is what you get from putting a second processor on the task. >>>even worse, i could argue that the interesting number is not 1.9 or 1.99 but >>>rather 2-x, with x now being how much inefficiency you get from the SMP search >>>overhead. and now we are talking about a factor of 10 in the result which can go >>>wrong. this is not a completely useless number either - obviously, an SMP search >>>which gets 1.95 on average is a MUCH better search than one which gets 1.8 on >>>average - even though it will not affect the end speed that much. but it is >>>obviously getting much closer to perfection than the 1.8 search. >>> >>>if you want to stay with your interpretation (which i think is a bad idea), then >>>you can still get nearly 10% error by taking 1.01 and 1.10 instead of your >>>numbers - not that that is very likely to happen :-) >>>whatever interpretation of your data you want to use, there is 1. no reason to >>>round the way you do, and 2. it is always better to keep a digit too much than >>>one too little :-) >>> >>>aloha >>> martin >> >> >>Here is the problem. I can run the _same_ position 4 times and get greater >>than 10% variance in the times. All using 4 cpus. All on a machine by >>themselves. Ugly, eh? And in that regard, .1 is not very significant, as >>I did mention in the paper... > >i understand the problem. so you would have to test with a very large number of >positions. i don't have the paper, so i have to speculate: but let's just say >you get 10% standard deviation on a single result if you repeat it (have you >ever tested this, i mean, really? not just on one position and 4 times, but on >100 positions 100 times or so?), and you have 24 positions, so your average >speedup only has an error of 2% on it. and that is smaller than 0.1 ... >anyway, all of this still does not make biased rounding (you even do it twice!) >a good practice :-) > >aloha > martin Let me remind you that I did not say we did the "rounding" for certain. I said "we certainly could have done it that way..." Without the code, and since neither of the two people that worked on the code (myself or my partner) remember, we won't ever know this... unfortunately. However, from the speedup results, I _suspect_ we didn't do any real rounding and used floating point instead. Not that that really matters a bit of course.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.