Author: Walter Koroljow
Date: 19:39:18 02/05/01
Go up one level in this thread
Thank you for the detailed reply. I didn't know what the real problem was. But maybe I can help reconcile the two cases you talked about and take a small chip out of the general case. As you did, let's forget about draws. Let's compare the probability of getting 60-40, say, with the stronger program to the probability of getting it with the weaker program. In general, a very good simple approximation to the answer is: Ratio = exp(8*eps*(x-N/2)) Where: x = program score, N = number of games played, eps = We - 0.5 where We = win expectancy. This approximation is excellent as long as eps is not too big and as long as N is on the order of 10 or more. Let us try a few cases: N = 100, x = 60, We = 0.55, eps = 0.05. Then Ratio = exp (8*0.05*(60-50)) = exp (4) = 54.6. But, with the usual approach, using the (60,40) binomial term instead, canceling some terms in the numerator and denominator, we get Ratio = (.55/.45)^20 = 55.33. A 1.3% error. But the approximation becomes extremely accurate as We approaches 0.5. Try: N=100, x = 60, We = 0.51, eps = .01. Ratio = exp(0.8) = 2.2255. Compare this to (.51/.49)^20 = 2.2258. This approximation means that two test results are equivalent as long as they have the same value of eps*(x-N/2), since then the chance of confusing the two programs is the same for the two results. So here are some equivalent test results with a probability of confusing the two programs at 8%: N x We 10 10 .56 20 15 .56 20 20 .53 60 40 .53 60 60 .51 100 60 .53 It is hard to measure small differences! For the record, the approximation is derived by approximating the binomial distribution of scores with a normal distribution and then using it to calculate the ratio of the two probabilities for getting the observed result with the two programs. I can supply more detail if anyone is interested. Good luck, Walter
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.