Author: Robert Hyatt
Date: 13:14:55 07/31/03
Go up one level in this thread
On July 30, 2003 at 17:21:10, Sune Fischer wrote: >On July 30, 2003 at 16:31:50, Angrim wrote: > >>On July 29, 2003 at 23:56:03, Robert Hyatt wrote: >> >>>You finally reached the ultimate conclusion. :) >>> >>>But to re-iterate another point, perft is a correctness benchmark _only_, as it >>>is defined. It is _not_ a performance benchmark. If it were, most of us would >>>write it differently, certainly using hashing to avoid traversing duplicate >>>parts of the tree searched due to transpositions. >> >>That would be monumentally stupid, unless the numbers were being used by >>a marketing department to sell the engine or something. If the goal is >>for the person who is coding the perft() to find out how his movegen, makemove, >>and unmakemove speed compares to others to get an idea of how much he >>could speed it up, or to compare the speed of two different versions of >>his own movegen, then there is no rational reason to cheat, since this >>only hurts himself. >>This is like someone designing a racecar trying to benchmark how fast >>their car is by putting it up on blocks and taking the wheels off, and >>timing how fast the axles rotate. >> I think that timing perft can be a useful benchmark, as long as you >>code perft as defined and visit each leaf node. If you use hash tables >>and avoid visiting the leaf nodes, then you have written something that >>outputs the same number as perft does, but it isn't the same function. > >Perft is dangerous to use as a benchmark, because it doesn't show you what you >think it shows. > >The idea is to peal off things like move ordering, extensions and evaluation, >and then time the raw movegen + makemove part. > >But if you consider the tree a real chessprogram has to search, you'll find that >it is completely different from the perft tree. >In perft you need to search *all* the moves that you generate, in a chess >program on average that's only ~3 moves you need to search, some you will get >directly from killer and hash tables with no generation at all. > >If you try and compare profiles of a perft run with a real search, you will see >that the perft run spends something like 85-90% in makemove, and only a fraction >on the generation. In a chess program this could easily be 50-50 (just throwing >out some numbers here). > >The conclusion is, that if you benchmark and optimize for perft, you're not >necessarily optimizing for the chessprogram. Suppose you find a way to make >makemove 15% faster and movegen 25% slower, in perft this will look like a >performance increase, but it won't be in a real program - necessarily (depends >on the profile). > >So it can hurt to do this, another example that can be misleading is incremental >move generators. This makes perft go slower but programs faster. > >You might also want to put some incremental stuff into the makemove or movegen >to speed up other parts of the program at the expense of the move stuff. > >So it doesn't make a lot of sense to compare programs here, you simple do not >compare what you want to compare. > >In summery, perft is a debug tool, not a benchmark :) > >-S. >>Angrim That's a good point. IE in Crafty, I generate captures and non-captures separately. Often the captures are good enough to produce a cutoff and the non-captures don't even get generated. But in perft, I produce all moves every time, even though that doesn't happen in a real chess game. If I were sloppier and produced all the moves at once, perft would actually run faster, but the program would run slower.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.