Author: Uri Blass
Date: 20:52:14 07/30/03
Go up one level in this thread
On July 30, 2003 at 23:25:14, Angrim wrote: >On July 30, 2003 at 17:21:10, Sune Fischer wrote: > >>On July 30, 2003 at 16:31:50, Angrim wrote: >> >>>On July 29, 2003 at 23:56:03, Robert Hyatt wrote: >>> >>>>You finally reached the ultimate conclusion. :) >>>> >>>>But to re-iterate another point, perft is a correctness benchmark _only_, as it >>>>is defined. It is _not_ a performance benchmark. If it were, most of us would >>>>write it differently, certainly using hashing to avoid traversing duplicate >>>>parts of the tree searched due to transpositions. >>> >>>That would be monumentally stupid, unless the numbers were being used by >>>a marketing department to sell the engine or something. If the goal is >>>for the person who is coding the perft() to find out how his movegen, makemove, >>>and unmakemove speed compares to others to get an idea of how much he >>>could speed it up, or to compare the speed of two different versions of >>>his own movegen, then there is no rational reason to cheat, since this >>>only hurts himself. >>>This is like someone designing a racecar trying to benchmark how fast >>>their car is by putting it up on blocks and taking the wheels off, and >>>timing how fast the axles rotate. >>> I think that timing perft can be a useful benchmark, as long as you >>>code perft as defined and visit each leaf node. If you use hash tables >>>and avoid visiting the leaf nodes, then you have written something that >>>outputs the same number as perft does, but it isn't the same function. >> >>Perft is dangerous to use as a benchmark, because it doesn't show you what you >>think it shows. >> >>The idea is to peal off things like move ordering, extensions and evaluation, >>and then time the raw movegen + makemove part. >> >>But if you consider the tree a real chessprogram has to search, you'll find that >>it is completely different from the perft tree. >>In perft you need to search *all* the moves that you generate, in a chess >>program on average that's only ~3 moves you need to search, some you will get >>directly from killer and hash tables with no generation at all. > >This is a valid point. Optimizing for perft alone without taking into >account how a real search behaves would have bad effects. > >>If you try and compare profiles of a perft run with a real search, you will see >>that the perft run spends something like 85-90% in makemove, and only a fraction >>on the generation. In a chess program this could easily be 50-50 (just throwing >>out some numbers here). > >Actually, half my time in perft is spent on InCheck calls. But I agree >with your point here, In a real search I call InCheck about 2x as often >as I call movegen, but in perft() I call it 30x as often. > >>The conclusion is, that if you benchmark and optimize for perft, you're not >>necessarily optimizing for the chessprogram. Suppose you find a way to make >>makemove 15% faster and movegen 25% slower, in perft this will look like a >>performance increase, but it won't be in a real program - necessarily (depends >>on the profile). > >optimizing only for perft wouldn't work well, you have to take into >account what sort of change you have made. But if you had made a >change which only affects your movegen, and get perft(6) 5% faster, this >is a very good hint that the change was an improvement. > >>So it can hurt to do this, another example that can be misleading is incremental >>move generators. This makes perft go slower but programs faster. >> >>You might also want to put some incremental stuff into the makemove or movegen >>to speed up other parts of the program at the expense of the move stuff. >> >>So it doesn't make a lot of sense to compare programs here, you simple do not >>compare what you want to compare. >> >>In summery, perft is a debug tool, not a benchmark :) >> >>-S. > >You made several good points. Optimizing a program for perft alone >would be a bad idea. perft tests your makemove/unmake speed much >more than a real search would. Not for me because I do not make the last move. I have a legal move generator so making the last move may lead to misleading results because when I make move I not only update the board array but update attack and pin information that are used to generate the legal moves of the next ply. However, perft still has uses for speed >testing I think, you just have to use some sense along with it. >If you just made your eval incremental, and perft slowed down, you >need to be smart enough not to get upset by that. On the other hand >if you just did a one line change to your makemove intended to speed >it up, and perft slowed down, you should worry. Of course but you should worry also if your program search less nodes per second. The main use of perft is for checking that you do not have a new bug and not for testing speed and there are other ways to test speed relative to your previous version. Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.