Author: Erik Bergren
Date: 09:37:05 08/08/03
Go up one level in this thread
On August 06, 2003 at 10:54:49, Erik Bergren wrote: >> >>Hi George. >>I have a question. I am not a mathematician and I'm sure many who visit here >>aren't either. Why is 600-700 games the magic number? Dann Corbit explained >>that mathematics will draw the line as to how many games it would take to show >>which program is stronger. If after 100 games one program is trounced could one >>not with reasonable assurance say one program plays chess better than the other? >> What would be the mathematical formula for wins and losses after 100 games to >>say with reasonable assurance that program A is stronger than program B? What >>if one program won 85 out of one hundred games with no only 5 draws? >>Thanks >. > > All "Probability Ratios" stay constant regardless of >the number of samples (such as 100 or 600). This must >be true simply because: time has no effect on probabilistic >results. There would be an exception to that if: >the programs were able to learn (but that effect >would not show up until a huge number of games were played). > As to your question of "assurance" that 85 wins out >of 100 shows one to be superior to the other: >Just model the problem with pennies: >assume an equal chance to a flip resulting in a heads up landing >of the penny. Thus getting 85 heads out of 100 would be >quite unlikely indeed ( just have your computer calculate >all possible out comes, and find the ratio of the total >of all of those, to those that have more than or equal to >85 heads out of 100). You will thus be shown that >one program not being better than the other (thus them being >equal in strength), after 85 wins out of 100, is >less likely than 1 in 1000000 ( I do not have a >computer with me to find the exact number). Your inverse question "How many games would it take" is ambiguous: it depends on the closeness of the opponents in rating, an infinitesimal rating difference between the two opponents would imply an infinite number of games to determine which of them is better. Here is the full equation and answer (after consulting my computer): I have been using the statistical equation that I used to answer the following post to determine if the frequency of wins vs. losses in an openning book is "convincing", Is my equation correct? Posted by Erik Bergren (Profile) on August 06, 2003 at 10:54:49: In Reply to: Re: Statistical Question posted by Dana Turnmire on August 05, 2003 at 12:11:09: > >Hi George. >I have a question. I am not a mathematician and I'm sure many who visit here >aren't either. Why is 600-700 games the magic number? Dann Corbit explained >that mathematics will draw the line as to how many games it would take to show >which program is stronger. If after 100 games one program is trounced could one >not with reasonable assurance say one program plays chess better than the other? > What would be the mathematical formula for wins and losses after 100 games to >say with reasonable assurance that program A is stronger than program B? What >if one program won 85 out of one hundred games with no only 5 draws? >Thanks . All "Probability Ratios" stay constant regardless of the number of samples (such as 100 or 600). This must be true simply because: time has no effect on probabilistic results. There would be an exception to that if: the programs were able to learn (but that effect would not show up until a huge number of games were played). As to your question of "assurance" that 85 wins out of 100 shows one to be superior to the other: Just model the problem with pennies: assume an equal chance to a flip resulting in a heads up landing of the penny. Thus getting 85 heads out of 100 would be quite unlikely indeed ( just have your computer calculate all possible out comes, and find the ratio of the total of all of those, to those that have more than or equal to 85 heads out of 100). You will thus be shown that one program not being better than the other (thus them being equal in strength), after 85 wins out of 100, is less likely than 1 in 1000000 ( I do not have a computer with me to find the exact number).Now I do: The equation is: integral (from x=85 to 100) of [(.5)^x*(.5)^(100-x)*(100!)/((x!)*(100-x)!)] divided by integral (from x=50 to 100) of [(.5)^x*(.5)^(100-x)*(100!)/((x!)*(100-x)!)] which equals 2^25. Thus the probability (or "strangeness") of a computer winning 85 (or higher) out of 100 games from one of equal strength ( or against an identical computer, say ) is one in 10^25. Thus such a result would conclusively show that one of those computers was better than it's opponent.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.