Computer Chess Club Archives




Subject: Re: New tool to estimate the statistical significance of match results

Author: Joachim Rang

Date: 11:21:00 07/17/04

Go up one level in this thread

Hi Remi,

that is exatly what I need right now, thank you very much.
Unfortunately the MonteCarlo calculation takes really long if I feed in the
result of two 400 games-gauntlets of two different version. I aborted after 10
minutes. Any idea how long it might take to compute on my Athlon xp 1540 Mhz? Is
ther a way to make it a bit less accurate but quicker?

regards Joachim

On July 17, 2004 at 10:07:36, Rémi Coulom wrote:

>I have made a new tool to estimate the likelihood that one program is better
>than another, based on game results against the same opponents. You can download
>it here:
>I have no evidence, but I expect its results to be more satisfactory than any
>result based on Elo theory. This tool was very useful to me during the
>preparation of the WCCC, to test differences between versions of TCB. I hope
>some of you will find it useful too.
>Here is a sample output of MonteCarlo.exe:
>  This program evaluates the likelihood that program A is better than
>program B, based on the result of two matches played against the same
>opponent (or set of opponents). The number of games played in each of
>these matches does not have to be the same. If playing against a set
>of opponents, the proportion of each opponent should be the same in each
>  The likelihood is estimated by Bayesian inference, assuming an uniform
>prior distribution of the probabilites of losing and winning.
>  The resulting integral is estimated with a Monte-Carlo method. It may
>take a long time to converge when the number of games is large (>100).
>The computation can be interrupted at any time with Ctrl-C.
>A wins   = 3
>A losses = 4
>A draws  = 5
>B wins   = 6
>B losses = 7
>B draws  = 8
>P(A>B) = 0.459296 (127000000 Iterations)

This page took 0.01 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.