Author: Don Dailey
Date: 22:18:10 06/27/98
Go up one level in this thread
On June 27, 1998 at 22:52:13, Robert Hyatt wrote: >On June 27, 1998 at 19:28:01, Mark Young wrote: > >> >>Are their any test suites that can give you an idea of how strong a program >>really is? The more I run test suites today, the more I am convinced that they >>are meaningless in gauging a program s strength > >no. several have tried, but running a suite, then fitting a bizarre function >to known ratings of programs that can be used to produce results for that >suite. But this doesn't mean a thing to *other* programs that are different >from the ones used to create the formula. > >Ask Don about Larry's formula from an old suite of his that he thought was >pretty good, *until* we ran the suite on Cray Blitz at Indianapolis ACM (I >think). It blew out the formula badly... and gave results that were just >about meaningless. because it solved almost everything in 0 seconds per >position... > >much better to do as you (and I and Bruce and others) do and play them on a >server against known humans and programs. That gives a much better feel than >a series of chess positions. I don't remember this incident but I've heard you refer to it a couple of times and do not doubt you. But I agree with your analysis that there are no good test suites that will tell you much about the strength of a chess program. I am almost (but not quite) of the opinion that tactics don't have much to do with the strength of a chess program any more, since they all seem adequate in this area! I haven't seen my program win a program tactically in a long time. It seems to get into won positions gradually and then it's over. When it loses it seems to be the same thing. A whole lot of games we've won seem to be because the other program just made some positional error, we've won a lot of lost games this way even. The extra depth helps a WHOLE lot, but I don't think a tactical problem set represents how games are won and lost, because as I've said, they always start from won positions. What they really seem to measure is if your program can win QUICK from a won position! I'm not sure how to capture what is really going on in computer chess with a test suite. Looking for standard good positional moves may be a small part of the picture but I think it has mostly to do with not making errors. Who can beat you if you make no errors? Depth helps a whole lot in computer chess and tactical sets provide a rough measurement of depth and so there is a corelation between this and speed although it is a very rough measurement. A relatively poor program can get a relatively good score on a tactical set and this can happen the other way around too. I do not think there are winning moves in chess, only losing ones. Or another way of saying this is that EVERY move must be a winning move. In a theoretical draw (like the opening position probably is) both players play winning moves by not ever getting into a lost position. Sometimes we say that move x is the winning move and give it an exclamation point! But we know this is really a human perception thing, it's another way of saying that the player was already winning but had to find a difficult move to preseve the win. THE winning move, really should mean the first move played after the opponent makes an error! And after this of course you must find a winning move on EVERY move after that. - Don
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.