Author: Robert Hyatt
Date: 10:16:17 05/07/04
Go up one level in this thread
On May 07, 2004 at 12:31:47, Anthony Cozzie wrote: >On May 07, 2004 at 04:42:55, martin fierz wrote: > >>On May 07, 2004 at 03:17:44, Gian-Carlo Pascutto wrote: >> >>>On May 06, 2004 at 19:03:48, martin fierz wrote: >>> >>>>point #3 is perhaps most important for the bob vs vincent duel: the standard >>>>error for a 4 CPU test run is on the order of 0.2. if vincent's tests were with >>>>a similarly small number of positions, then the differences measured in these >>>>experiments (2.8 / 3.0 / 3.1) are statistically insignificant, and the whole >>>>argument is pointless :-) >>> >>>Not recessarily - disabling nullmove produced a result with the standard errors >>>halved in my results. That would still allow a significant conclusion. >>> >>>If I assume your 0.2 is a 2SD number (95%), your results are compatible and >>>running a non-nullmove test could still produce the same result. If 0.2 is a >>>1SD number then for some reason your results were much more variable then mine. >>> >>> n speedup error (1SD) >>>------------------------------------------ >>>Nullmove 38 2.82 +- 0.101 >>>No-nullmove 39 3.07 +- 0.056 >>> >>>-- >>>GCP >> >>a repeat of "my" numbers >> >>4 CPUs: >>3.15 +- 0.15 >>3.29 +- 0.20 >>3.06 +- 0.12 >>3.19 +- 0.13 > >Something is amiss: > >Martin: 3.17 +- 0.17 >GCP: 2.82 +- 0.10 > >Which means that from GCP's perspective martin's result is impossible (> 3 SD >away!). > >Which leaves us with 3 explainations: > >1. Crafty has changed It has changed, but not significantly within the SMP code. Eval and search have changed some of course. But these will most likely affect the speedup in random and disconnected ways... >2. Crafty has a different speedup on Opteron vs P3 Not by my testing. Things are within reasonable variance. IE I got 3.1 +/- n on my quad xeon when I first posted the results here. That hasn't changed. I have seen 2.5 and 3.5 and even beyond over a set of positions... >3. Speedup is not normally distributed. > I think it is probably normally distributed on a single position. And a different normal distribution over the set of positions. So there is a good bit of variance. Using one run on 24-30 positions to establish the variance will not produce a really good number... >I am guessing 3. > >anthony
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.