Author: Don Dailey
Date: 22:47:15 01/14/98
Go up one level in this thread
Hi Bruce, I use the method of waiting for the iteration to finish but my main motivation for doing this has nothing to do with waiting for it to change it's mind. I just believe the results tend to be less random this way because when you see the right move is a random luck thing because of where it happens to appear in the list. But actually the effect is not so large as to be that big a concern. I don't think what I do is any better than what you do though and I'm always looking for better methods. I don't run too many problem sets and the ones I do run tend to be the type where you don't change your mind once you see the right idea. Your testing methodology seems quite sound to me. - Don On January 14, 1998 at 14:59:40, Bruce Moreland wrote: > >On January 14, 1998 at 14:09:11, David Fotland wrote: > >>I haven't done any testing of Blitzen using test suites. I've just >>played about 35 games with rated players. > >Figure out how to connect it to a chess server, and you can get this >many games in a few hours, while you sleep. > >>Can someone tell me how to get some test suites so I can try some >>automated testing, please. I tried WAC003, since it was posted >>here, and solve it quickly, and get to 8 ply with about 550k nodes. > >ftp://ics.onenet.net/pub/chess/Tests/ > >There are some in here that are interesting. > >If you want to do some surfing, you can start with: > >http://www.clark.net/pub/pribut/chess.html > >You can find more suites in various places by cruising the links there. > >>When solving these problems, do you count it as solved at the first >>iteration you get the right answer, or do you require that answer to >>hold over several iterations, or do you require a change in score >>at the first iteration with the right answer? Sometimes the search >>changes its mind back and forth between two moves over several >>iterations. > >Ideally what you want to do is eyeball your answer and see if it >satisfies you. This takes more time than anyone has, especially in >large suites. > >The next best case, in my opinion, is to record a solution as the time >taken to find and hold the solution until the end of the test period. > >If you find it in 8 seconds, something else looks better at 32 seconds, >and you find it again at 72 seconds and hold the answer until you >terminate the test, I would score this as 72 seconds. > >Some programs let you save time by specifying that a solution is >considered valid if it is found and held for some time, for instance, >until the end of the ply after it was found. I don't agree with this >method, since it has been my experience that programs will switch away >from a solution often enough for this to be reflected in the results. >Count up the number of times you find a solution in ply 4, hold it until >the end of ply 5, and lose it in ply 6, and you'll see what I mean. > >The point at which a position is solved is the point beyond which the >program would play the solution move if it ran out of time. Some others >record the time when the iteration that found the move ends, and others >might record the initial target time that would be required to find the >move (which might be less than the amount of time taken to find the >move), but I think this first method makes the most sense. > >bruce
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.