Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Proving something is better

Author: Uri Blass

Date: 14:24:12 12/18/02

Go up one level in this thread


On December 18, 2002 at 16:42:33, Brian Richardson wrote:

>On December 18, 2002 at 16:20:16, Robert Hyatt wrote:
>>
>>That's not what I was talking about.  There are two ways you can compare
>>two chess algorithms, logically.
>>
>>1.  time until it finds the solution move with the right score.
>
>There is a 3rd and IMHO variation, let's call it 1A) find the right move
>for *whatever* reason (score).  Yes, of course, plain 1 (right score) would be
>best, but in real games, how much should we care?
>
>I have been struggling with this with just plain old WAC300.
>Is 299 in 10 minutes "better" than 295 in 2 minutes in terms of real game
>performance?  It seems to vary, depending on bullet (no), blitz (maybe), or
>(fairly long) standard time controls (yes).
>
>I agree with Bruce that testing methodologies should be discussed and hopefully
>developed.  My hunch is this is the *real* advantage that the commercial
>programs have worked out--much better testing.
>
>Right now for me, a "good" test is about 2 weeks of play on ICC for Tinker.
>Yes, I do "quick and dirty" WAC and other benchmark tests, but only to see if
>something seems broken.  Yet, even after playing hundreds of games during 2
>weeks, the rating is sometimes random (+/- 50), based on the mix of opponents.
>
>Brian

I think that you can use free winboard programs to do a better test because you
can control the number of games that you play against every opponent.

You can start with testing with very fast time control of only some seconds per
game(I catched a bug when movei declared a wrong mate when it played a game
against gerbil with 2 seconds per game,without showing thinking and with
animation off).

I use gerbil as an opponet at 1-10 seconds per game and I get matchs at 1-10
seconds per game when I type /mg 40 /tc 0:01/debug to get a match of 40 games at
1 second per game under winboard.

Unfortunately I do not know a way to play automatically 10 matches at different
time control.

If someone has an idea how can I can run the 400 games when I sleep then it may
productive for me.

Today I need to type manually under additional options
/mg 40 /tc 0:01/debug
After the match is finished I need to type again
/mg 40 /tc 0:02/debug
and later again
/mg 40 /tc 0:03/debug

I cannot do it when I sleep.
I wish I could do it by only one type but unfortunately I need to repeat this
process 10 times.

Some comments about the games:
1)the best result so far is of the latest tested version and it got slightly
more than 280 out of 400(I do not have the page that I wrote the results here
and I do not remember the exact result)

2)Games with 1 second per game are often won on time but not always(Gerbil lose
more than movei on time)
At longer time control big majority of the games are not won on time.

3)I decide that a win on time is always a win for the side that won even if it
is a draw by the rules because I have no time to analyze the games to correct
the results.

4)I use ponder off and movei of today with ponder on needs longer time control
not to lose on time(I can fix it but it was not important for me).

5)The last result at 5 minutes per game of a previous version against Gerbil was
better and it got 32-5 with 3 draws(remember that movei can play stupid opening
like 1.a4 or 1.h4 in the condition that I describe but it usually does not help
gerbil).

I did that match only to check that I probably have no new serious bug.
I believe that the difference relative to gerbil is bigger at longer time
control and 2 or 3 seconds per game is the time that gerbil likes most against
movei because it can avoid losing on time and it is still fast time control.

6)Movei does not use the opponent time and I know that it is a mistake.
In games with ponder off when the opponent is in big time trouble using more
time can be a good idea because the only chance of the opponent is to find a
mate and using more time may prevent the opponent to find a mate and it does not
take risks of losing on time.

I believe that only using the opponent time may give movei a bonus of more than
10 out of 400 in the games against gerbil and maybe it can be even enough to get
total result of more than 300 out of 400.

Uri



This page took 0.15 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.