Author: Tom Kerrigan
Date: 10:28:45 05/23/00
Go up one level in this thread
On May 23, 2000 at 12:27:40, Bruce Moreland wrote: >On May 23, 2000 at 09:10:32, Dan Homan wrote: > >>On May 23, 2000 at 03:59:50, Tom Kerrigan wrote: >> >>>Hi guys, >>> >>>Is there a good way to measure search efficiency? >>> >>>In the past, I've gone through test suite logs and compared nodes/ply for each >>>problem by hand. This is obviously undesirable. :) >>> >>>Is there some way I can get a good "magic number" to indicate how efficient my >>>search is? Is branching factor good or bad? Is there something similar but >>>better? >>> >>>Thanks, >>>Tom >> >>I don't know if this is what you are looking for, but a year or so ago on this >>group, Bruce Moreland suggested comparing graphs of solution times and ply >>depth reached as a measure of search improvements. I liked this idea but >>decided to just use mean ply depth and root mean square (RMS) solution time >>instead. >> >>The mean ply depth for a test suite is nice because it tells me about how deep >>my program is going in a fixed period of time (I throw Mates out of the average >>because they cut the search time short). The RMS solution time is also nice >>because I want my program to find the best move as quickly as possible. I use >>the RMS solution time because it weights up the problems that take the longest >>to solve. >> >>To optimize my search I try to get the best mean ply depth and RMS solution time >>compromise on test suites. If I turn up extensions, I usually can reduce the >>RMS solution time significantly (particularly on a tactical suite), but the >>mean ply depth goes down significantly as well. If I increase pruning, my >>mean ply depth goes up significantly, but my RMS solution time also goes up >>(and some previously solved problems go unsolved). >> >>I think the two measures are simple and off-setting which makes them useful >>to me. >> >> - Dan > >If I want to check a new search heuristic I run the following test. In my >logfiles I include time taken to find a given solution. I have a program that >will go collect these times and dump them out. I don't just display the number, >I display the number correct after each second. So I know how many took one >second to solve, two seconds to solve, etc. > >This gives me more information that just dumping out the final number. I have >sometimes seen cases where the number of total solutions isn't the most >important information. Sometimes the program will pick up a lot in the first >few seconds and fall off, and sometimes the reverse is true. > >Also, I can use the whole set of data to get a better idea of the actual >improvement in tactical speed. If I get 400 right in the first case, and 420 in >the second case, that indicates a good version, but if I can also see that the >second version got 400 right in half the time it took the first version, it >indicates that I doubled tactical speed. With this kind of tool you can >actually use WAC data for something, not that I try. I usually use ECM. > >If I want to track a performance change I use another program. In my logfile I >know how long it took for the program to complete any given search depth for >each problem. I can compare two versions by figuring out how long it took for >each of them to complete the highest depth that both of them completed, for a >given problem, and I have something that dumps this information in such a way >that I can read it. This is usually what I do by hand but it makes me a little nervous. It's possible that a program searches to 7 ply quickly but 8 ply slowly. So if you make the program a little slower, you don't finish 8 ply, so you compare the 7 ply scores, and all of a sudden the program appears faster (even though it's actually slower). What I've started to do is search all the BK positions to n ply and count the total nodes. I think this is a good idea. -Tom
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.