Author: Dave Gomboc
Date: 12:52:17 09/02/99
Go up one level in this thread
On September 02, 1999 at 13:04:49, Heiko Mikala wrote: >On September 01, 1999 at 23:28:18, Dave Gomboc wrote: > >>On September 01, 1999 at 19:01:32, Mark Young wrote: >> >>>On September 01, 1999 at 18:26:48, Heiko Mikala wrote: >>> >>[snip] >>>>game/60', 2 * Cyrix/IBM 6x86MX PR-300 >>>> >>>> 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 >>>>1 Hiarcs 7.32 ½ 0 ½ 1 ½ 1 0 1 ½ 1 1 1 1 1 1 1 1 13.0/17 >>>>2 CSTal II ½ 1 ½ 0 ½ 0 1 0 ½ 0 0 0 0 0 0 0 0 4.0/17 >>> > >[snip] > >>>>And - as always - the same old comment: 17 games are by far not enough to say >>>>which program is stronger. You need at least 60-80 games for that! >> >>With a score this lopsided, 17 games won't be "far not enough". > > >Sure?!? > >Here are some results of test matches, that my own chess engine played against >another well known engine. Note, that I didn't explicitly search for good >examples but simply picked a few matches at random: > >1.) > 1234567890123456789012345678901234567890 >HMChess 1100½10001½1101101½½000½100½111101111001 22.0/40 >XXX 0011½01110½0010010½½111½011½000010000110 18.0/40 > >Look at games 39 to 47. If you only take these games, the score for HMChess >would be 8-1, a very decisive win. But the real result of this match was only a >22-18 ... > > >2.) > 123456789012345678901234567890123456 >HMChess 11½100100111111001½01½1001½11½001111 --> >XXX 00½011011000000110½10½0110½00½110000 > > 789012345678901234567890123456789012 > --> 0½0101½1000011½11½1½1½111½0000½00100 39.5/73 > 1½1010½0111100½00½0½0½000½1111½11011 32.5/73 > >Take games 10 to 15 for example. 6 wins in a row, but a fairly equal end result >for the match. > >The first 15 games of this match look very similar to my posted results for the >match CSTal II - Hiarcs, don't they? > >3.) > 123456789012345678901234567890123456 >HMChess 011111100011110110½0101½1100111½½100 --> >XXX 100000011100001001½1010½0011000½½011 > > 789012345678901234567890123456789012 > --> 0½01½½101100½0½0½001½10001110011½0½1 38.5/73 > 1½10½½010011½1½1½110½01110001100½1½0 33.5/73 > >Take games 2 to 7 for an example of 6 wins in a row. > >Endresult only 38.5-33.5 ... > >4.) > 123456789012345678901234567890123456 >HMChess 10110110110½011100100000110101001000 --> >XXX 01001001001½100011011111001010110111 > > 789012345678901234567890123456789012 > --> 1001011½½00110½111½10½100111½0111100 37.5/72 > 0110100½½11001½000½01½011000½1000011 34.5/72 > >And this is an example, where it was the other way round (maybe the best example >for what I want to say): > >If you only take the games 27 to 34 you will find a result of 7-1 for the other >engine. But in the end, HMChess won the match instead, with a score of 37.5-34.5 >!! > >-------------------------------------------------------------------------- > >So, I repeat myself and state again: 17 games ar by far not enough to draw *any* >conclusions. Although, as I said, I have the *feeling* that Hiarcs would win a >longer match. > >By the way, I originally didn't join this thread to defend CSTal. I'm not >affiliated to the project or the programmers in any way, I simply happen to own >the program and I like it's style. > >And yes (this is for Mark), I really tend to be a bean counter in normal life. >My goal is to make my own engine as strong as possible, so I have to be. And >sometimes I would like to know which commercial program is strongest. I'm crazy >sometimes. ;-) > >Greetings, > >Heiko. I don't think that you addressed my point, namely: Less games are required to conclude with a certain confidence that one program is better than another when the results are lopsided than when they are not. Dave
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.