Author: Gian-Carlo Pascutto
Date: 11:57:46 05/05/04
Go up one level in this thread
On May 05, 2004 at 11:18:52, Robert Hyatt wrote: >>av. speedup: 3.65 >>standard deviation of sample: 0.31 >>standard error of average 0.064 >> >>so: average speedup(N=4) = 3.65 +- 0.07 would be a nice way to put this. > >Where does the standard error come from? The same place they all come from I'd assume :) std error = 1/sqrt(n) * std_dev(sample) He explained what the sample is. >Are you looking at the speedups for >two positions and using the difference (summed over all positions) as the error? > That's one kind of error. The other is the non-repeatible error which is the >real problem that needs addressing. If I run the _same_ test again, rather than >3.65 it might produce 3.3 or 3.9 which is the real problem... He already factored in the non-reproducibility by taking a sample over different positions. That should do, unless you are expecting the differences in speedups over the same positions to be bigger than the differences between different positions. (Now that would make no sense) >I sent both the log file sothey can confirm both my 3.1 and GCP's 2.8. I don't have any log from you. >Yes, although I think the basic proof is trivial. On related positions you >simply search deeper due to the hash table effects. Schaeffer and others have >repeatedly found (myself included I failed to add) that deeper searches make the >search more efficient. But doing this test is harder. IE it isn't reasonable >to search to "fixed depth" for each position as that is not how it works in a >real game and it can skew the times somewhat... adding yet another bit of >variability... One of my main complaints with this methodology is quite simply that your measurements are not independent. Hence any analysis like this looks like very hazy business, notably averaging speedups over positions that are NOT independent. Comparison to illustrate this effect: We do a running match. We start at the same time. I run faster than you, what you run in 3 minutes I do in 2. We arrive at checkpoint 1, I at 2 minutes, you at 3. I have a speedup of 1.5 over you. You note this down. Of course, by the time you have arrived, I have a 1 minute headstart over you to the next checkpoint. We arrive at checkpoint 2, I only need 1 more minute, you need 3 again. I have a speedup of 3 over you. You note this down. Of course, by the time you have arrived, I have a 2 minute headstart over you to the next checkpoint. We arrive at checkpoint 3, I'm already there because of my 2 minutes headstart :) You need 3 more minutes. My speedup over you is infinite. You note this down. My speedup is the average of inf, 3 and 1.5, whatever that might be :) This is what you did in the paper, and I don't think it's correct. -- GCP
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.