Author: Vasik Rajlich
Date: 06:55:55 09/02/05
Go up one level in this thread
On September 01, 2005 at 13:52:09, Peter Berger wrote:
>On August 31, 2005 at 08:53:56, Vasik Rajlich wrote:
>
>>On August 31, 2005 at 06:22:49, Peter Berger wrote:
>>
>>>On August 31, 2005 at 04:52:11, Vasik Rajlich wrote:
>>>
>>>>On August 30, 2005 at 12:27:52, Peter Berger wrote:
>>>>
>>>>>On August 30, 2005 at 12:21:20, Maurizio De Leo wrote:
>>>>>
>>>>>>
>>>>>>>Under valid and controlled conditions it still seems logical to me to stop a
>>>>>>>test after a 5-0 result and conclude that the winning program is probably the
>>>>>>>stronger one.
>>>>>>
>>>>>>>>I don't put much credence in any result of less than 30 games.
>>>>>>>>After 30 games, then you get a lot more plausibility.
>>>>>>
>>>>>>>You didn't give any reason for this, so I don't understand. A 6-0 says more
>>>>>>>about engine strength than the above match result with over 100000 games.
>>>>>>
>>>>>>Dann is right, I think.
>>>>>>The confidence interval calculation assumes that the score of a game is a
>>>>>>statistic variable with a mean value between 1 and -1 (function of the Elo
>>>>>>difference between the programs) and a standard deviation. Then if the
>>>>>>experiments are independent, the sum of the points will approximate the product
>>>>>>(mean*number of games) with a smaller standard deviation the more the games are.
>>>>>>With enough games the "confidence" will get to 95% when the performance
>>>>>>difference between the two programs is more than 3 standard deviations.
>>>>>>However this assumes a normal distribution. The assumption can be made for any
>>>>>>repeated statistical variable as long as the experiments are independent and
>>>>>>"enough". This "enough" is indeed expressed in most statistics books as 30.
>>>>>>
>>>>>>Maurizio
>>>>>
>>>>>Please have a look at "WhoisBest.zip" at Rémi Coulom's Home Page:
>>>>>http://remi.coulom.free.fr/. It includes a little paper Whoisbest.pdf on
>>>>>"Statistical Significance of a Match" , with a very straightforward mathematical
>>>>>proof that for example the number of draws is irrelevant to conclude who is
>>>>>better in a chessmatch .
>>>>>
>>>>>Peter
>>>>
>>>>It's not that simple, due to the nature of chess.
>>>>
>>>>In chess, a match result of 2-0 with 0 draws is less significant than a match
>>>>result of 2-0 with 8 draws.
>>>>
>>>>WhoIsBest makes the assumption that draws are independent events - that is, that
>>>>wins, losses and draws each come with some independent probability. In fact, in
>>>>a +2 -0 =8 result, the chance is that the side with the +2 was "stronger" in the
>>>>draws - ie. closer to winning. Chess has this phenomenon where the stronger side
>>>>tries to break through the draw barrier, and sometimes cannot.
>>>>
>>>>Of course to model this mathematically would be a huge mess.
>>>>
>>>>Vas
>>>
>>>No, that's a misunderstanding.
>>>
>>>The only assumption that is made is that the results get drawn independently
>>>from an unknown probability distribution.
>>>
>>>So it doesn't matter *at all* how drawish chess itself is e.g. . And the result
>>>will be the same whether the game is tic-tac-toe, checkers or chess.
>>>
>>>Unless you want to argue that there should be a distinction between drawn games,
>>>depending on how close one side got to winning. But that's a completely
>>>different topic.
>>>
>>>Peter
>>
>>Ok - consider the following scenario:
>>
>>Two players are playing basketball. The stronger player has some >50% chance to
>>score each basket. The game ends when one player scores 50 points. Once the game
>>is finished, a win by a margin of under 25 points is declared a draw, while a
>>win by >25 points is declared a win.
>>
>>The question is: in this case, is a 2-0 result with 8 draws more significant
>>than 2-0 with 0 draws?
>>
>>Vas
>
>No, it isn't more significant on answering the question who is the better
>player.
>
>Peter
It's more complicated than that.
I just did the following (very simplistic) simulation:
White has 1% chance to win:
Black has 20% chance to win:
250,000 iterations
White won 2-0 with 0 draws: 10,000 times (exactly :-))
Black won with 0 draws: 38
White won 2-0 with 8 draws: 13,584 times
Black won with 8 draws: 30 times
I don't think I have to explain this further. If you want to play around, code
is appended (.NET form, C# w/ designer code - I guess I should say very ugly).
Consider another scenario.
You make a very small tweak to your eval, run 5 games, and the score is 5-0 for
the new version. WhoIsBest says 95%+ chance that it's an improvement. What is
the actual chance (at that point)? More like 51%.
Vas
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
namespace Tester_Dialogs
{
public partial class Statistics : Form
{
public Statistics()
{
InitializeComponent();
}
private void label6_Click(object sender, EventArgs e)
{
}
private void Calculate_Odds_Click(object sender, EventArgs e)
{
int iterations = (int)this.Iterations_Value.Value;
int scenario1 = 0;
int scenario2 = 0;
int scenario3 = 0;
int scenario4 = 0;
Random rand = new Random(1); // seeded random number
int stronger_perc = (int)this.Stronger_Numeric.Value;
int weaker_perc = (int)this.Weaker_Numeric.Value;
int win_goal = (int)this.Wins_Numeric.Value;
int draws_goal = (int)this.Draws_Numeric.Value;
for (int i = 0; i < iterations; i++)
{
int stronger_wins = 0;
int weaker_wins = 0;
int draws = 0;
while (true)
{
if (false)
{
int str_score = 0;
int wkr_score = 0;
while (true)
{
int random_number = rand.Next(0, 100);
if (random_number < stronger_perc)
str_score++;
else
wkr_score++;
if (str_score == 50 || wkr_score == 50)
break;
}
if (Math.Abs(str_score - wkr_score) < 25)
draws++;
else if (str_score > wkr_score)
stronger_wins++;
else
weaker_wins++;
}
else
{
int random_number = rand.Next(0, 100);
if (random_number < stronger_perc)
stronger_wins++;
else if (random_number < stronger_perc + weaker_perc)
weaker_wins++;
else
draws++;
}
if (stronger_wins == win_goal || weaker_wins == win_goal)
break;
}
if (stronger_wins == win_goal && weaker_wins == 0 && draws == 0)
{
scenario1++;
}
if (stronger_wins == 0 && weaker_wins == win_goal && draws == 0)
{
scenario2++;
}
if (stronger_wins == win_goal && weaker_wins == 0 && draws ==
draws_goal)
{
scenario3++;
}
if (stronger_wins == 0 && weaker_wins == win_goal && draws ==
draws_goal)
{
scenario4++;
}
} // iterations
this.Scenario1_Value.Text = scenario1.ToString();
this.Scenario2_Value.Text = scenario2.ToString();
this.Scenario3_Value.Text = scenario3.ToString();
this.Scenario4_Value.Text = scenario4.ToString();
}
}
}
namespace Tester_Dialogs
{
partial class Statistics
{
/// <summary>
/// Required designer variable.
/// </summary>
private System.ComponentModel.IContainer components = null;
/// <summary>
/// Clean up any resources being used.
/// </summary>
/// <param name="disposing">true if managed resources should be
disposed; otherwise, false.</param>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}
#region Windows Form Designer generated code
/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
this.Weaker_Label = new System.Windows.Forms.Label();
this.Stronger_Label = new System.Windows.Forms.Label();
this.Weaker_Numeric = new System.Windows.Forms.NumericUpDown();
this.Stronger_Numeric = new System.Windows.Forms.NumericUpDown();
this.Calculate_Odds = new System.Windows.Forms.Button();
this.Wins_Label = new System.Windows.Forms.Label();
this.Wins_Numeric = new System.Windows.Forms.NumericUpDown();
this.Scenario1_Label = new System.Windows.Forms.Label();
this.Draws_Label = new System.Windows.Forms.Label();
this.Draws_Numeric = new System.Windows.Forms.NumericUpDown();
this.Scenario1_Value = new System.Windows.Forms.Label();
this.Scenario2_Label = new System.Windows.Forms.Label();
this.Scenario3_Label = new System.Windows.Forms.Label();
this.Scenario4_Label = new System.Windows.Forms.Label();
this.Scenario2_Value = new System.Windows.Forms.Label();
this.Scenario3_Value = new System.Windows.Forms.Label();
this.Scenario4_Value = new System.Windows.Forms.Label();
this.Iterations_Label = new System.Windows.Forms.Label();
this.Iterations_Value = new System.Windows.Forms.NumericUpDown();
this.label1 = new System.Windows.Forms.Label();
this.numericUpDown1 = new System.Windows.Forms.NumericUpDown();
((System.ComponentModel.ISupportInitialize)(this.Weaker_Numeric)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.Stronger_Numeric)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.Wins_Numeric)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.Draws_Numeric)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.Iterations_Value)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.numericUpDown1)).BeginInit();
this.SuspendLayout();
//
// Weaker_Label
//
this.Weaker_Label.Location = new System.Drawing.Point(13, 13);
this.Weaker_Label.Name = "Weaker_Label";
this.Weaker_Label.Size = new System.Drawing.Size(119, 23);
this.Weaker_Label.TabIndex = 0;
this.Weaker_Label.Text = "Weaker Winning %:";
//
// Stronger_Label
//
this.Stronger_Label.Location = new System.Drawing.Point(13, 36);
this.Stronger_Label.Name = "Stronger_Label";
this.Stronger_Label.Size = new System.Drawing.Size(119, 23);
this.Stronger_Label.TabIndex = 1;
this.Stronger_Label.Text = "Stronger Winning %:";
//
// Weaker_Numeric
//
this.Weaker_Numeric.Location = new System.Drawing.Point(153, 9);
this.Weaker_Numeric.Name = "Weaker_Numeric";
this.Weaker_Numeric.Size = new System.Drawing.Size(120, 20);
this.Weaker_Numeric.TabIndex = 2;
this.Weaker_Numeric.Value = new decimal(new int[] {
2,
0,
0,
0});
//
// Stronger_Numeric
//
this.Stronger_Numeric.Location = new System.Drawing.Point(153, 33);
this.Stronger_Numeric.Name = "Stronger_Numeric";
this.Stronger_Numeric.Size = new System.Drawing.Size(120, 20);
this.Stronger_Numeric.TabIndex = 3;
this.Stronger_Numeric.Value = new decimal(new int[] {
50,
0,
0,
0});
//
// Calculate_Odds
//
this.Calculate_Odds.Location = new System.Drawing.Point(88, 199);
this.Calculate_Odds.Name = "Calculate_Odds";
this.Calculate_Odds.Size = new System.Drawing.Size(75, 23);
this.Calculate_Odds.TabIndex = 4;
this.Calculate_Odds.Text = "Calculate";
this.Calculate_Odds.Click += new
System.EventHandler(this.Calculate_Odds_Click);
//
// Wins_Label
//
this.Wins_Label.Location = new System.Drawing.Point(13, 82);
this.Wins_Label.Name = "Wins_Label";
this.Wins_Label.Size = new System.Drawing.Size(119, 23);
this.Wins_Label.TabIndex = 5;
this.Wins_Label.Text = "Wins:";
//
// Wins_Numeric
//
this.Wins_Numeric.Location = new System.Drawing.Point(153, 81);
this.Wins_Numeric.Name = "Wins_Numeric";
this.Wins_Numeric.Size = new System.Drawing.Size(120, 20);
this.Wins_Numeric.TabIndex = 6;
this.Wins_Numeric.Value = new decimal(new int[] {
2,
0,
0,
0});
//
// Scenario1_Label
//
this.Scenario1_Label.Location = new System.Drawing.Point(12, 242);
this.Scenario1_Label.Name = "Scenario1_Label";
this.Scenario1_Label.Size = new System.Drawing.Size(211, 23);
this.Scenario1_Label.TabIndex = 7;
this.Scenario1_Label.Text = "stronger wins, 0 losses, 0 draws ct:";
this.Scenario1_Label.TextAlign =
System.Drawing.ContentAlignment.MiddleRight;
//
// Draws_Label
//
this.Draws_Label.Location = new System.Drawing.Point(13, 105);
this.Draws_Label.Name = "Draws_Label";
this.Draws_Label.Size = new System.Drawing.Size(119, 23);
this.Draws_Label.TabIndex = 8;
this.Draws_Label.Text = "Draws:";
//
// Draws_Numeric
//
this.Draws_Numeric.Location = new System.Drawing.Point(153, 106);
this.Draws_Numeric.Maximum = new decimal(new int[] {
1000,
0,
0,
0});
this.Draws_Numeric.Name = "Draws_Numeric";
this.Draws_Numeric.Size = new System.Drawing.Size(120, 20);
this.Draws_Numeric.TabIndex = 9;
this.Draws_Numeric.Value = new decimal(new int[] {
8,
0,
0,
0});
//
// Scenario1_Value
//
this.Scenario1_Value.Location = new System.Drawing.Point(229, 242);
this.Scenario1_Value.Name = "Scenario1_Value";
this.Scenario1_Value.Size = new System.Drawing.Size(211, 23);
this.Scenario1_Value.TabIndex = 10;
this.Scenario1_Value.Text = "-";
this.Scenario1_Value.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// Scenario2_Label
//
this.Scenario2_Label.Location = new System.Drawing.Point(13, 265);
this.Scenario2_Label.Name = "Scenario2_Label";
this.Scenario2_Label.Size = new System.Drawing.Size(211, 23);
this.Scenario2_Label.TabIndex = 11;
this.Scenario2_Label.Text = "weaker wins, 0 losses, 0 draws ct:";
this.Scenario2_Label.TextAlign =
System.Drawing.ContentAlignment.MiddleRight;
//
// Scenario3_Label
//
this.Scenario3_Label.Location = new System.Drawing.Point(13, 288);
this.Scenario3_Label.Name = "Scenario3_Label";
this.Scenario3_Label.Size = new System.Drawing.Size(211, 23);
this.Scenario3_Label.TabIndex = 12;
this.Scenario3_Label.Text = "stronger wins, 0 losses, x draws ct:";
this.Scenario3_Label.TextAlign =
System.Drawing.ContentAlignment.MiddleRight;
//
// Scenario4_Label
//
this.Scenario4_Label.Location = new System.Drawing.Point(13, 311);
this.Scenario4_Label.Name = "Scenario4_Label";
this.Scenario4_Label.Size = new System.Drawing.Size(211, 23);
this.Scenario4_Label.TabIndex = 13;
this.Scenario4_Label.Text = "weaker wins, 0 losses, x draws ct:";
this.Scenario4_Label.TextAlign =
System.Drawing.ContentAlignment.MiddleRight;
//
// Scenario2_Value
//
this.Scenario2_Value.Location = new System.Drawing.Point(230, 265);
this.Scenario2_Value.Name = "Scenario2_Value";
this.Scenario2_Value.Size = new System.Drawing.Size(211, 23);
this.Scenario2_Value.TabIndex = 14;
this.Scenario2_Value.Text = "-";
this.Scenario2_Value.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
this.Scenario2_Value.Click += new
System.EventHandler(this.label6_Click);
//
// Scenario3_Value
//
this.Scenario3_Value.Location = new System.Drawing.Point(229, 288);
this.Scenario3_Value.Name = "Scenario3_Value";
this.Scenario3_Value.Size = new System.Drawing.Size(211, 23);
this.Scenario3_Value.TabIndex = 15;
this.Scenario3_Value.Text = "-";
this.Scenario3_Value.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// Scenario4_Value
//
this.Scenario4_Value.Location = new System.Drawing.Point(230, 311);
this.Scenario4_Value.Name = "Scenario4_Value";
this.Scenario4_Value.Size = new System.Drawing.Size(211, 23);
this.Scenario4_Value.TabIndex = 16;
this.Scenario4_Value.Text = "-";
this.Scenario4_Value.TextAlign =
System.Drawing.ContentAlignment.MiddleLeft;
//
// Iterations_Label
//
this.Iterations_Label.Location = new System.Drawing.Point(13, 128);
this.Iterations_Label.Name = "Iterations_Label";
this.Iterations_Label.Size = new System.Drawing.Size(119, 23);
this.Iterations_Label.TabIndex = 17;
this.Iterations_Label.Text = "Iterations:";
//
// Iterations_Value
//
this.Iterations_Value.Location = new System.Drawing.Point(153, 130);
this.Iterations_Value.Maximum = new decimal(new int[] {
10000000,
0,
0,
0});
this.Iterations_Value.Name = "Iterations_Value";
this.Iterations_Value.Size = new System.Drawing.Size(120, 20);
this.Iterations_Value.TabIndex = 18;
this.Iterations_Value.Value = new decimal(new int[] {
100000,
0,
0,
0});
//
// label1
//
this.label1.Location = new System.Drawing.Point(321, 187);
this.label1.Name = "label1";
this.label1.Size = new System.Drawing.Size(119, 23);
this.label1.TabIndex = 19;
this.label1.Text = "Stronger Winning %:";
//
// numericUpDown1
//
this.numericUpDown1.Location = new System.Drawing.Point(387, 242);
this.numericUpDown1.Name = "numericUpDown1";
this.numericUpDown1.Size = new System.Drawing.Size(120, 20);
this.numericUpDown1.TabIndex = 20;
this.numericUpDown1.Value = new decimal(new int[] {
50,
0,
0,
0});
//
// Statistics
//
this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(603, 417);
this.Controls.Add(this.numericUpDown1);
this.Controls.Add(this.label1);
this.Controls.Add(this.Iterations_Value);
this.Controls.Add(this.Iterations_Label);
this.Controls.Add(this.Scenario4_Value);
this.Controls.Add(this.Scenario3_Value);
this.Controls.Add(this.Scenario2_Value);
this.Controls.Add(this.Scenario4_Label);
this.Controls.Add(this.Scenario3_Label);
this.Controls.Add(this.Scenario2_Label);
this.Controls.Add(this.Scenario1_Value);
this.Controls.Add(this.Draws_Numeric);
this.Controls.Add(this.Draws_Label);
this.Controls.Add(this.Scenario1_Label);
this.Controls.Add(this.Wins_Numeric);
this.Controls.Add(this.Wins_Label);
this.Controls.Add(this.Calculate_Odds);
this.Controls.Add(this.Stronger_Numeric);
this.Controls.Add(this.Weaker_Numeric);
this.Controls.Add(this.Stronger_Label);
this.Controls.Add(this.Weaker_Label);
this.Name = "Statistics";
this.Text = "Statistics";
((System.ComponentModel.ISupportInitialize)(this.Weaker_Numeric)).EndInit();
((System.ComponentModel.ISupportInitialize)(this.Stronger_Numeric)).EndInit();
((System.ComponentModel.ISupportInitialize)(this.Wins_Numeric)).EndInit();
((System.ComponentModel.ISupportInitialize)(this.Draws_Numeric)).EndInit();
((System.ComponentModel.ISupportInitialize)(this.Iterations_Value)).EndInit();
((System.ComponentModel.ISupportInitialize)(this.numericUpDown1)).EndInit();
this.ResumeLayout(false);
}
#endregion
private System.Windows.Forms.Label Weaker_Label;
private System.Windows.Forms.Label Stronger_Label;
private System.Windows.Forms.NumericUpDown Weaker_Numeric;
private System.Windows.Forms.NumericUpDown Stronger_Numeric;
private System.Windows.Forms.Button Calculate_Odds;
private System.Windows.Forms.Label Wins_Label;
private System.Windows.Forms.NumericUpDown Wins_Numeric;
private System.Windows.Forms.Label Scenario1_Label;
private System.Windows.Forms.Label Draws_Label;
private System.Windows.Forms.NumericUpDown Draws_Numeric;
private System.Windows.Forms.Label Scenario1_Value;
private System.Windows.Forms.Label Scenario2_Label;
private System.Windows.Forms.Label Scenario3_Label;
private System.Windows.Forms.Label Scenario4_Label;
private System.Windows.Forms.Label Scenario2_Value;
private System.Windows.Forms.Label Scenario3_Value;
private System.Windows.Forms.Label Scenario4_Value;
private System.Windows.Forms.Label Iterations_Label;
private System.Windows.Forms.NumericUpDown Iterations_Value;
private System.Windows.Forms.Label label1;
private System.Windows.Forms.NumericUpDown numericUpDown1;
}
}
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.