Author: Uri Blass
Date: 14:24:12 12/18/02
Go up one level in this thread
On December 18, 2002 at 16:42:33, Brian Richardson wrote: >On December 18, 2002 at 16:20:16, Robert Hyatt wrote: >> >>That's not what I was talking about. There are two ways you can compare >>two chess algorithms, logically. >> >>1. time until it finds the solution move with the right score. > >There is a 3rd and IMHO variation, let's call it 1A) find the right move >for *whatever* reason (score). Yes, of course, plain 1 (right score) would be >best, but in real games, how much should we care? > >I have been struggling with this with just plain old WAC300. >Is 299 in 10 minutes "better" than 295 in 2 minutes in terms of real game >performance? It seems to vary, depending on bullet (no), blitz (maybe), or >(fairly long) standard time controls (yes). > >I agree with Bruce that testing methodologies should be discussed and hopefully >developed. My hunch is this is the *real* advantage that the commercial >programs have worked out--much better testing. > >Right now for me, a "good" test is about 2 weeks of play on ICC for Tinker. >Yes, I do "quick and dirty" WAC and other benchmark tests, but only to see if >something seems broken. Yet, even after playing hundreds of games during 2 >weeks, the rating is sometimes random (+/- 50), based on the mix of opponents. > >Brian I think that you can use free winboard programs to do a better test because you can control the number of games that you play against every opponent. You can start with testing with very fast time control of only some seconds per game(I catched a bug when movei declared a wrong mate when it played a game against gerbil with 2 seconds per game,without showing thinking and with animation off). I use gerbil as an opponet at 1-10 seconds per game and I get matchs at 1-10 seconds per game when I type /mg 40 /tc 0:01/debug to get a match of 40 games at 1 second per game under winboard. Unfortunately I do not know a way to play automatically 10 matches at different time control. If someone has an idea how can I can run the 400 games when I sleep then it may productive for me. Today I need to type manually under additional options /mg 40 /tc 0:01/debug After the match is finished I need to type again /mg 40 /tc 0:02/debug and later again /mg 40 /tc 0:03/debug I cannot do it when I sleep. I wish I could do it by only one type but unfortunately I need to repeat this process 10 times. Some comments about the games: 1)the best result so far is of the latest tested version and it got slightly more than 280 out of 400(I do not have the page that I wrote the results here and I do not remember the exact result) 2)Games with 1 second per game are often won on time but not always(Gerbil lose more than movei on time) At longer time control big majority of the games are not won on time. 3)I decide that a win on time is always a win for the side that won even if it is a draw by the rules because I have no time to analyze the games to correct the results. 4)I use ponder off and movei of today with ponder on needs longer time control not to lose on time(I can fix it but it was not important for me). 5)The last result at 5 minutes per game of a previous version against Gerbil was better and it got 32-5 with 3 draws(remember that movei can play stupid opening like 1.a4 or 1.h4 in the condition that I describe but it usually does not help gerbil). I did that match only to check that I probably have no new serious bug. I believe that the difference relative to gerbil is bigger at longer time control and 2 or 3 seconds per game is the time that gerbil likes most against movei because it can avoid losing on time and it is still fast time control. 6)Movei does not use the opponent time and I know that it is a mistake. In games with ponder off when the opponent is in big time trouble using more time can be a good idea because the only chance of the opponent is to find a mate and using more time may prevent the opponent to find a mate and it does not take risks of losing on time. I believe that only using the opponent time may give movei a bonus of more than 10 out of 400 in the games against gerbil and maybe it can be even enough to get total result of more than 300 out of 400. Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.