# Computer Chess Club Archives

## Messages

### Subject: Re: Testing the reliability of forward pruning

Author: Robert Hyatt

Date: 17:39:40 05/18/03

Go up one level in this thread

```On May 18, 2003 at 04:28:08, scott farrell wrote:

>On May 16, 2003 at 10:42:25, Robert Hyatt wrote:
>
>>On May 15, 2003 at 21:47:20, Jon Dart wrote:
>>
>>>Ernst Heinz did this by using standard test suites, for example Win at Chess, or
>>>ECM (Encyclopedia of Chess Middlegames). He found that the solve rate didn't
>>>really change much with forward pruning on, but the number of nodes searched for
>>>a fixed ply depth decreased 20-50% (this is from the chapter on AEL pruning in
>>>his book Scaleable Search in Computer Chess). He also used other testing
>>>methods, including game play, as detailed in the book.
>>
>>
>>I don't think this methodology is reasonable.
>>
>>For example, suppose your forward-pruning speeds you up by a factor of
>>four.  Comparing same-search-depth runs means the FP version will move
>>in 1/4th the time of the non-FP version.  Suppose the FP version does
>>worse on three positions?  But if you run it for 4x longer, so that it
>>has the same time limit as the non-FP version, the three positions are now
>>"back to normal"???
>>
>>I think the _right_ way to test is with a fixed time limit so that a version
>>that can go deeper will go deeper.  After all, that is the _purpose_ of FP
>>in the first place...
>
>maybe you can run the non-fp version, and time how long it takes to find the
>solution. Rhen allow a maximum time control for the fp version equal to the
>solve time - if it solves in less, success, if it doesnt solve in time =
>failure.
>
>Scott

That's my point.  I use a "sum of squares" approach.  I search to solution,
square the time and sum all such times for all the positions.  I compare this
to the sum of squared times for the other version, and the version with the
"smaller" sum of squares is better...

>
>>
>>
>>
>>
>>>
>>>--Jon
>>>
>>>On May 15, 2003 at 18:17:09, Russell Reagan wrote:
>>>
>>>>I would like to know how to test whether or not a forward pruning method is
>>>>reliable.
>>>>
>>>>I have one idea to test when and if a method is reliable, and I'd like to know
>>>>if it's a good idea or not, and also what other methods might be used to test
>>>>the reliability of forward pruning methods.
>>>>
>>>>My idea requires a collection of games, and two versions of a program. One
>>>>version would have forward pruning turned off, and the other would have it
>>>>turned on. You would feed each version of the program the same game, and let
>>>>each do a search on the initial position to the same fixed depth. If both
>>>>versions report the same move and score, and the version using forward pruning
>>>>had a lower time to depth, then the forward pruning is reliable (so far). If the
>>>>version using forward pruning reported different results, then the forward
>>>>pruning method is not reliable for this type of position. You make the next move
>>>>in the game, and repeat the search and compare the results for each position in
>>>>the game. Then you repeat the process for each game.
>>>>
>>>>When I think about testing the reliability of null-move using this method, I
>>>>think the test would do well. I would expect the test to tell us that in most
>>>>positions, null-move is reliable, and I would expect it to fail for some endgame
>>>>positions, and so this test would tell us that null-move was good forward
>>>>pruning, but to turn it off in the endgame (or detect zugzwang, or however you
>>>>choose to guard against it). I haven't had time to test this though, since I
>>>>just thought of it and I'm not at home.
>>>>
>>>>I am basing all of this on the assumption that the strength forward pruning
>>>>provides is not that it finds better moves at the same depth, but that it
>>>>finishes searching a particular depth in a shorter amount of time, allowing the
>>>>search to go deeper, which is where the added strength comes from. Is this
>>>>correct?
>>>>