Computer Chess Club Archives




Subject: Re: Value of playing different versions of a program against each other

Author: Dann Corbit

Date: 14:03:01 01/06/03

Go up one level in this thread

On January 06, 2003 at 16:56:35, Tom King wrote:

>Hi all,
>What do people think about playing different versions of your program against
>each other as a way of testing?
>I'm playing around with it right now, between v0.07 and a newer version of my
>program. The newer version is winning handsomely: +24,=18,-10.
>This implies a reasonably impressive increase in strength, almost 100 ELO. Ok,
>ok, it's a small sample, so the margin of error could be big.
>However, my gut feel is that playing different versions of your programs tends
>to overstate the strength differences. What do people think?

That test demonstrates exactly what it measures:
Win expectancy against previous versions of your own program.

If you want to know win expectancy against other programs, you will have to test
it separately.

On the other hand, there is probably going to be some correlation between your
new program clubbing the old ones and how it fares against other programs.  On
the other hand, you won't have any idea what the correlation is until you test

This page took 0.07 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.