Author: Christophe Theron
Date: 20:40:50 05/23/00
Go up one level in this thread
On May 23, 2000 at 12:27:40, Bruce Moreland wrote:
>On May 23, 2000 at 09:10:32, Dan Homan wrote:
>
>>On May 23, 2000 at 03:59:50, Tom Kerrigan wrote:
>>
>>>Hi guys,
>>>
>>>Is there a good way to measure search efficiency?
>>>
>>>In the past, I've gone through test suite logs and compared nodes/ply for each
>>>problem by hand. This is obviously undesirable. :)
>>>
>>>Is there some way I can get a good "magic number" to indicate how efficient my
>>>search is? Is branching factor good or bad? Is there something similar but
>>>better?
>>>
>>>Thanks,
>>>Tom
>>
>>I don't know if this is what you are looking for, but a year or so ago on this
>>group, Bruce Moreland suggested comparing graphs of solution times and ply
>>depth reached as a measure of search improvements. I liked this idea but
>>decided to just use mean ply depth and root mean square (RMS) solution time
>>instead.
>>
>>The mean ply depth for a test suite is nice because it tells me about how deep
>>my program is going in a fixed period of time (I throw Mates out of the average
>>because they cut the search time short). The RMS solution time is also nice
>>because I want my program to find the best move as quickly as possible. I use
>>the RMS solution time because it weights up the problems that take the longest
>>to solve.
>>
>>To optimize my search I try to get the best mean ply depth and RMS solution time
>>compromise on test suites. If I turn up extensions, I usually can reduce the
>>RMS solution time significantly (particularly on a tactical suite), but the
>>mean ply depth goes down significantly as well. If I increase pruning, my
>>mean ply depth goes up significantly, but my RMS solution time also goes up
>>(and some previously solved problems go unsolved).
>>
>>I think the two measures are simple and off-setting which makes them useful
>>to me.
>>
>> - Dan
>
>If I want to check a new search heuristic I run the following test. In my
>logfiles I include time taken to find a given solution. I have a program that
>will go collect these times and dump them out. I don't just display the number,
>I display the number correct after each second. So I know how many took one
>second to solve, two seconds to solve, etc.
>
>This gives me more information that just dumping out the final number. I have
>sometimes seen cases where the number of total solutions isn't the most
>important information. Sometimes the program will pick up a lot in the first
>few seconds and fall off, and sometimes the reverse is true.
>
>Also, I can use the whole set of data to get a better idea of the actual
>improvement in tactical speed. If I get 400 right in the first case, and 420 in
>the second case, that indicates a good version, but if I can also see that the
>second version got 400 right in half the time it took the first version, it
>indicates that I doubled tactical speed. With this kind of tool you can
>actually use WAC data for something, not that I try. I usually use ECM.
>
>If I want to track a performance change I use another program. In my logfile I
>know how long it took for the program to complete any given search depth for
>each problem. I can compare two versions by figuring out how long it took for
>each of them to complete the highest depth that both of them completed, for a
>given problem, and I have something that dumps this information in such a way
>that I can read it.
>
>The same program will tell me if there were any node-count disparities, so if I
>make a change that is a pure performance change, one that shouldn't change the
>tree at all, I get a warning if I changed the tree, so I can track it down.
>
>Anyone who doesn't have this last capability is a dummy.
>
>bruce
Yes, I have this too. When I change something, even a single line, I run a test
and look at the total number of positions seen. If it is different, either I
know why (the change in tree size was expected) and it's OK, or I don't know why
and it's likely a bug.
The next step in this case is to use my tree dump utility. It allows me to
compare the trees searched by two different versions, and to point me exactly
where the searches forked. Then it is very easy to find what the problem is.
It has been especially useful when I ported my code from GCC to VC. The number
of nodes computed by the 2 versions were different, and without this tool it
would have taken me several days to find what the problem was. With the tool I
have been able to fix it in 1 hour.
Christophe
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.