Author: Bruce Moreland
Date: 11:59:40 01/14/98
Go up one level in this thread
On January 14, 1998 at 14:09:11, David Fotland wrote: >I haven't done any testing of Blitzen using test suites. I've just >played about 35 games with rated players. Figure out how to connect it to a chess server, and you can get this many games in a few hours, while you sleep. >Can someone tell me how to get some test suites so I can try some >automated testing, please. I tried WAC003, since it was posted >here, and solve it quickly, and get to 8 ply with about 550k nodes. ftp://ics.onenet.net/pub/chess/Tests/ There are some in here that are interesting. If you want to do some surfing, you can start with: http://www.clark.net/pub/pribut/chess.html You can find more suites in various places by cruising the links there. >When solving these problems, do you count it as solved at the first >iteration you get the right answer, or do you require that answer to >hold over several iterations, or do you require a change in score >at the first iteration with the right answer? Sometimes the search >changes its mind back and forth between two moves over several >iterations. Ideally what you want to do is eyeball your answer and see if it satisfies you. This takes more time than anyone has, especially in large suites. The next best case, in my opinion, is to record a solution as the time taken to find and hold the solution until the end of the test period. If you find it in 8 seconds, something else looks better at 32 seconds, and you find it again at 72 seconds and hold the answer until you terminate the test, I would score this as 72 seconds. Some programs let you save time by specifying that a solution is considered valid if it is found and held for some time, for instance, until the end of the ply after it was found. I don't agree with this method, since it has been my experience that programs will switch away from a solution often enough for this to be reflected in the results. Count up the number of times you find a solution in ply 4, hold it until the end of ply 5, and lose it in ply 6, and you'll see what I mean. The point at which a position is solved is the point beyond which the program would play the solution move if it ran out of time. Some others record the time when the iteration that found the move ends, and others might record the initial target time that would be required to find the move (which might be less than the amount of time taken to find the move), but I think this first method makes the most sense. bruce
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.