Author: Sune Fischer
Date: 14:21:10 07/30/03
Go up one level in this thread
On July 30, 2003 at 16:31:50, Angrim wrote: >On July 29, 2003 at 23:56:03, Robert Hyatt wrote: > >>You finally reached the ultimate conclusion. :) >> >>But to re-iterate another point, perft is a correctness benchmark _only_, as it >>is defined. It is _not_ a performance benchmark. If it were, most of us would >>write it differently, certainly using hashing to avoid traversing duplicate >>parts of the tree searched due to transpositions. > >That would be monumentally stupid, unless the numbers were being used by >a marketing department to sell the engine or something. If the goal is >for the person who is coding the perft() to find out how his movegen, makemove, >and unmakemove speed compares to others to get an idea of how much he >could speed it up, or to compare the speed of two different versions of >his own movegen, then there is no rational reason to cheat, since this >only hurts himself. >This is like someone designing a racecar trying to benchmark how fast >their car is by putting it up on blocks and taking the wheels off, and >timing how fast the axles rotate. > I think that timing perft can be a useful benchmark, as long as you >code perft as defined and visit each leaf node. If you use hash tables >and avoid visiting the leaf nodes, then you have written something that >outputs the same number as perft does, but it isn't the same function. Perft is dangerous to use as a benchmark, because it doesn't show you what you think it shows. The idea is to peal off things like move ordering, extensions and evaluation, and then time the raw movegen + makemove part. But if you consider the tree a real chessprogram has to search, you'll find that it is completely different from the perft tree. In perft you need to search *all* the moves that you generate, in a chess program on average that's only ~3 moves you need to search, some you will get directly from killer and hash tables with no generation at all. If you try and compare profiles of a perft run with a real search, you will see that the perft run spends something like 85-90% in makemove, and only a fraction on the generation. In a chess program this could easily be 50-50 (just throwing out some numbers here). The conclusion is, that if you benchmark and optimize for perft, you're not necessarily optimizing for the chessprogram. Suppose you find a way to make makemove 15% faster and movegen 25% slower, in perft this will look like a performance increase, but it won't be in a real program - necessarily (depends on the profile). So it can hurt to do this, another example that can be misleading is incremental move generators. This makes perft go slower but programs faster. You might also want to put some incremental stuff into the makemove or movegen to speed up other parts of the program at the expense of the move stuff. So it doesn't make a lot of sense to compare programs here, you simple do not compare what you want to compare. In summery, perft is a debug tool, not a benchmark :) -S. >Angrim
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.