Author: Robert Hyatt
Date: 12:43:06 05/05/04
Go up one level in this thread
On May 05, 2004 at 14:57:46, Gian-Carlo Pascutto wrote: >On May 05, 2004 at 11:18:52, Robert Hyatt wrote: > >>>av. speedup: 3.65 >>>standard deviation of sample: 0.31 >>>standard error of average 0.064 >>> >>>so: average speedup(N=4) = 3.65 +- 0.07 would be a nice way to put this. >> >>Where does the standard error come from? > >The same place they all come from I'd assume :) > >std error = 1/sqrt(n) * std_dev(sample) > >He explained what the sample is. > >>Are you looking at the speedups for >>two positions and using the difference (summed over all positions) as the error? >> That's one kind of error. The other is the non-repeatible error which is the >>real problem that needs addressing. If I run the _same_ test again, rather than >>3.65 it might produce 3.3 or 3.9 which is the real problem... > >He already factored in the non-reproducibility by taking a sample over >different positions. > >That should do, unless you are expecting the differences in speedups over the >same positions to be bigger than the differences between different positions. >(Now that would make no sense) > >>I sent both the log file sothey can confirm both my 3.1 and GCP's 2.8. > >I don't have any log from you. I sent it to you. And to vincent at the _same_ time. I can give you the exact date and contents of the email if you want... > >>Yes, although I think the basic proof is trivial. On related positions you >>simply search deeper due to the hash table effects. Schaeffer and others have >>repeatedly found (myself included I failed to add) that deeper searches make the >>search more efficient. But doing this test is harder. IE it isn't reasonable >>to search to "fixed depth" for each position as that is not how it works in a >>real game and it can skew the times somewhat... adding yet another bit of >>variability... > >One of my main complaints with this methodology is quite simply that your >measurements are not independent. Hence any analysis like this looks like >very hazy business, notably averaging speedups over positions that are NOT >independent. So how do you answer the question I was asked? "What kind of speedup do you get in real games?" Not "what kind of speedup do you get on test positions?" So. You have the question. You have my methodology to attempt to answer it. It is not acceptable to change the question. It is not acceptable to answer a different question. So give me a methodology that is not "hazy business" to answer _that_ specific question... > >Comparison to illustrate this effect: > >We do a running match. > >We start at the same time. I run faster than you, what you run in 3 minutes I do >in 2. > >We arrive at checkpoint 1, I at 2 minutes, you at 3. I have a speedup of 1.5 >over you. You note this down. Of course, by the time you have arrived, I have >a 1 minute headstart over you to the next checkpoint. I didn't do any such measurements. Times were _always_ started at the same point and ended at the same point... > >We arrive at checkpoint 2, I only need 1 more minute, you need 3 again. I have >a speedup of 3 over you. You note this down. Of course, by the time you have >arrived, I have a 2 minute headstart over you to the next checkpoint. > >We arrive at checkpoint 3, I'm already there because of my 2 minutes headstart >:) You need 3 more minutes. My speedup over you is infinite. >You note this down. > >My speedup is the average of inf, 3 and 1.5, whatever that might be :) > >This is what you did in the paper, and I don't think it's correct. > I did nothing of the kind in the paper, so I _really_ have no idea what you are talking about here, nor of what paper you read. Maybe you read the one Vincent claims I wrote about Crafty? My test positions were run in the normal way, but with a correctly pre-loaded hash table at the start. Nothing more, nothing less. The only thing different between running my test and the normal BK test was the positions. Which were related rather than being random positions. Any other "differences" are imaginary, not real... >-- >GCP
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.