Author: martin fierz
Date: 03:03:21 05/06/04
Go up one level in this thread
On May 05, 2004 at 11:02:08, Matthew Hull wrote: >On May 05, 2004 at 10:41:38, Rolf Tueschen wrote: > >>On May 05, 2004 at 10:29:48, Matthew Hull wrote: >> >>>On May 05, 2004 at 10:25:23, Matthew Hull wrote: >>> >>>>On May 05, 2004 at 04:44:19, martin fierz wrote: >>>> >>>>>On May 04, 2004 at 21:06:30, Robert Hyatt wrote: >>>>> >>>>>>On May 04, 2004 at 18:21:07, martin fierz wrote: >>>>>> >>>>>>>On May 04, 2004 at 13:44:21, Robert Hyatt wrote: >>>>>>> >>>>>>>>On May 04, 2004 at 10:49:42, martin fierz wrote: >>>>>>>> >>>>>>>>>On May 04, 2004 at 07:32:01, Rolf Tueschen wrote: >>>>>>>>> >>>>>>>>>>On May 04, 2004 at 07:11:15, martin fierz wrote: >>>>>>>>>> >>>>>>>>>>>On May 03, 2004 at 22:50:58, Robert Hyatt wrote: >>>>>>>>>>> >>>>>>>>>>>>If you recall, I _have_ given some error estimates in the past. Remember the >>>>>>>>>>>>wildly varying speedup numbers I showed you the first time this issue came up? >>>>>>>>>>> >>>>>>>>>>>i recall that you gave wildly varying speedup numbers, and an explanation for >>>>>>>>>>>why this happens. i don't recall a real error estimate, but that can be either >>>>>>>>>>>because >>>>>>>>>>>-> you gave one and i didn't see it >>>>>>>>>>>-> you gave one, i saw it and forgot >>>>>>>>>>>-> you didn't give one at all >>>>>>>>>>> >>>>>>>>>>>so... what kind of numbers would you give if you were pressed? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>Isn't it impolite to imply the third option if Bob JUST said that he did give >>>>>>>>>>some? >>>>>>>>> >>>>>>>>>no - asking questions always has to be allowed among scientists. forbidding to >>>>>>>>>ask questions is the hallmark of religious fanatics and fascists... but i >>>>>>>>>digress :-) >>>>>>>>>bob says he gave numbers, which he did. but IIRC, he never gave an error >>>>>>>>>estimate. so i am allowed to ask for it, and it is not at all impolite to do so. >>>>>>>>>what he did show is the speedup in about 30 different positions, which could >>>>>>>>>vary wildly depending on the position. >>>>>>>>> >>>>>>>>>i don't know why you think you have to stand up and defend bob every time >>>>>>>>>somebody says something about him you don't like. just leave that up to him. he >>>>>>>>>can take it :-) >>>>>>>>> >>>>>>>>>cheers >>>>>>>>> martin >>>>>>>> >>>>>>>> >>>>>>>>I wasn't offended. I hope my answer was ok. >>>>>>> >>>>>>>i didn't think you'd be offended, and your answer was ok, but...why don't you >>>>>>>take N (preferably N>>30...) positions and compute the standard deviation of >>>>>>>your speedup numbers, and the standard deviation of the average speedup? you can >>>>>>>still discuss the meaning of this, but at least you have an error margin you can >>>>>>>attach to your speedup. i don't see anything wrong with that!? even if the >>>>>>>probability distribution is obviously not a normal distribution, you can >>>>>>>probably approximate it as such, and get an idea of it's width from these >>>>>>>numbers. >>>>>>> >>>>>>>>This is not an easy question to deal with. >>>>>>> >>>>>>>>IE if you take the standard deviation of a set of random numbers between >>>>>>>>0 and N what do you get? That is what the speedup numbers look like for some >>>>>>>>positions. For others the speedup is a near-perfect constant value. Add some >>>>>>>>perfect constants plus some randomly distributed values and exactly what does >>>>>>>>the SD show? :) >>>>>>> >>>>>>>i don't quite understand your question. if you take enough positions, then you >>>>>>>will get something sensible, i would think. if you doubt this, you can take e.g. >>>>>>>10'000 sequential positions from crafty's ICC log, and bunch them together in >>>>>>>groups of 1000, and compute average speedup + stdev-of-average-speedup for each >>>>>>>of the bunches. i can't imagine that you get 10 wildly differing values, as your >>>>>>>statement above suggests. >>>>>>> >>>>>>>cheers >>>>>>> martin >>>>>> >>>>>> >>>>>>It isn't so easy to get speedup. IE how would I take a position that took X >>>>>>seconds with 2 or 4 cpus and compute the 1 cpu time? Think about it carefully >>>>>>and you will see the problem. How to get the 1-cpu test case to have a properly >>>>>>loaded hash table, killer move table, history table, etc, before starting the >>>>>>search??? >>>>> >>>>>i don't understand this part at all. run the exact same test on a 1 CPU machine, >>>>>and then on an N-CPU machine. >>>>>the reason i don't understand this at all is that all the details you are >>>>>talking about are irrelevant. i want to know what happens if i run the same test >>>>>positions on a 1 CPU box or on a N CPU box. this is easy to answer, because it >>>>>can be determined experimentally. whether or not there are philosophical issues >>>>>as those you raise above can be discussed, but it doesn't stop you from getting >>>>>your number... >>>>> >>>>> >>>>>>The best bet is to take N positions where N is large. But then that is not the >>>>>>same as what happens in real games where the positions are connected via info >>>>>>passed from search to search in the hash table. >>>>> >>>>>not true if you use 1000 positions from crafty's ICC log file, where the exact >>>>>same thing happens, because these are also positions from real games, connected >>>>>to each other. >>>> >>>> >>>>The only way I can see how your idea would work is if crafty played a series of >>>>games searching to a fixed depth with a single processor. Then take the >>>>positions from those games and set them up in the order they occurred and time >>>>crafty's response to those positions to the same depth, but with 2 processors, >>>>then do it all over again with 3 processors, etc. That way for each run, the >>>>effects of cache relavency are not lost as would be the case in disconnected >>>>test set positions. >>>> >>>>Then you might have an idea of an expected speedup in actual games, rather than >>>>from disconnected test positions. >>>> >>>>That is a much more involved and time consuming test. I don't think he could >>>>have afforded that kind of CPU time on a CRAY. >>> >>>Not to mention that you would need to time the opponents response exactly for >>>each test run to get the effects of pondering on cache as well, I would assume. >>>Thats an extra complication. >> >>Yes, but this ius exactly what Bob is saying all the time, but what Martin >>doesn't follow... > > >Besides which, pondering makes no sense if you are searching to a fixed depth, >which means it's kind of impossible to get the numbers as they affect a real >game, because you can't replicate the effects of pondering. > >I think Martin has not thought this through. no, definitely i didn't think it through. but look: you are saying you can't measure anything because it's too complicated. the real world is also complicated. people study systems which are complicated all the time by studying "model systems". i.e. systems which they can control better. physicists study atoms in ultra high vacuum, and their findings have something to do with the real world outside of the ultra high vacuum chamber. what i'm trying to say is that you can test certain things, and *perhaps* make some conclusions about them. i'm not saying: "this will conclusively explain everything". the approach "i can't measure anything because it's too complicated" is even worse than my approach IMO. you can go and measure the difference between unrelated and related positions in a test such as i suggested. BTW, bob seemingly actually already did something similar to this test, so perhaps my suggestions aren't all that ridiculous?! then you can *still* discuss whether pondering and all the other stuff that goes on which you don't have in the test is important. all that this test does is that it's a part of the puzzle. if it clearly shows that there is a significant difference between related & unrelated positions, this would greatly bolster the hypothesis that this is true in real games. even though the test is admittedly not perfect. perhaps bob can think of a better test? in any case, making an experiment is much more convincing to me than waving your hands and saying "i know this will be like so and so because blah blah whatever". if it is that clear, it will show up in a mock experiment too... cheers martin
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.