Computer Chess Club Archives


Search

Terms

Messages

Subject: why test suites are weak

Author: Jay Scott

Date: 08:46:38 03/06/00

Go up one level in this thread



On March 06, 2000 at 04:42:21, Roger wrote:

>Maybe the test suites are all interbred to
>some degree. Or perhaps the test suites don't test what is really important in
>chess knowledge.

Test suites have holes. For example, they tend to leave out "boring"
positions with several good moves, no matter how easy it is to misunderstand
them.

But even if you corrected that, there's still the problem that the
suite won't be matched to each different program's style of play. In a real
game a program can try to reach positions that it knows how to play well,
but in a suite it has to take what comes. If a program plays to its own
strengths it may play better than its test results say, and if it doesn't
then it may play worse.

To get a relevant test suite, test each program on positions that occur
in its own games. This is easily done by having it play games. :-)



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.