Author: Bruce Moreland
Date: 09:27:40 05/23/00
Go up one level in this thread
On May 23, 2000 at 09:10:32, Dan Homan wrote: >On May 23, 2000 at 03:59:50, Tom Kerrigan wrote: > >>Hi guys, >> >>Is there a good way to measure search efficiency? >> >>In the past, I've gone through test suite logs and compared nodes/ply for each >>problem by hand. This is obviously undesirable. :) >> >>Is there some way I can get a good "magic number" to indicate how efficient my >>search is? Is branching factor good or bad? Is there something similar but >>better? >> >>Thanks, >>Tom > >I don't know if this is what you are looking for, but a year or so ago on this >group, Bruce Moreland suggested comparing graphs of solution times and ply >depth reached as a measure of search improvements. I liked this idea but >decided to just use mean ply depth and root mean square (RMS) solution time >instead. > >The mean ply depth for a test suite is nice because it tells me about how deep >my program is going in a fixed period of time (I throw Mates out of the average >because they cut the search time short). The RMS solution time is also nice >because I want my program to find the best move as quickly as possible. I use >the RMS solution time because it weights up the problems that take the longest >to solve. > >To optimize my search I try to get the best mean ply depth and RMS solution time >compromise on test suites. If I turn up extensions, I usually can reduce the >RMS solution time significantly (particularly on a tactical suite), but the >mean ply depth goes down significantly as well. If I increase pruning, my >mean ply depth goes up significantly, but my RMS solution time also goes up >(and some previously solved problems go unsolved). > >I think the two measures are simple and off-setting which makes them useful >to me. > > - Dan If I want to check a new search heuristic I run the following test. In my logfiles I include time taken to find a given solution. I have a program that will go collect these times and dump them out. I don't just display the number, I display the number correct after each second. So I know how many took one second to solve, two seconds to solve, etc. This gives me more information that just dumping out the final number. I have sometimes seen cases where the number of total solutions isn't the most important information. Sometimes the program will pick up a lot in the first few seconds and fall off, and sometimes the reverse is true. Also, I can use the whole set of data to get a better idea of the actual improvement in tactical speed. If I get 400 right in the first case, and 420 in the second case, that indicates a good version, but if I can also see that the second version got 400 right in half the time it took the first version, it indicates that I doubled tactical speed. With this kind of tool you can actually use WAC data for something, not that I try. I usually use ECM. If I want to track a performance change I use another program. In my logfile I know how long it took for the program to complete any given search depth for each problem. I can compare two versions by figuring out how long it took for each of them to complete the highest depth that both of them completed, for a given problem, and I have something that dumps this information in such a way that I can read it. The same program will tell me if there were any node-count disparities, so if I make a change that is a pure performance change, one that shouldn't change the tree at all, I get a warning if I changed the tree, so I can track it down. Anyone who doesn't have this last capability is a dummy. bruce
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.