Author: Dann Corbit
Date: 10:57:44 09/12/01
Go up one level in this thread
On September 12, 2001 at 03:51:07, Bernhard Bauer wrote: >On September 11, 2001 at 14:04:31, Dann Corbit wrote: > >>On September 11, 2001 at 08:34:53, Bernhard Bauer wrote: >> >>>On September 11, 2001 at 07:44:37, Gian-Carlo Pascutto wrote: >>> >>>>On September 11, 2001 at 03:39:52, Bernhard Bauer wrote: >>>> >>>>>BTW testing with WAC will not lead to any progress! >>>> >>>>Yes and no. It is good for tuning extensions, as it's >>>>all simple tactics. >>>> >>>>-- >>>>GCP >>> >>>Yes it's mostly about simple tactics - mate in 3,4,5 moves. >>>Tuning your program for wac will not help for real live positions, >>>so it' worthless, but some programmers are so much used to it, they can't stop >>>it. And it's so fast. But it's useless. You'd better take 5 well known positions >>>and run your program against them. >> >>There are very few close checkmates. >>Programs that score 290+ in WAC at 5 seconds per position are all powerhouses >>Programs that score below 250 at 5 seconds all suck. >>Draw your own conclusions. > >Due to your comments I had a closer look at WAC. >Running yace on WAC for 1 sec on a P3-450MHz gives 269 solved which is 90%. >Yace found at that run 71 mates which is 24%. >There are >16 mate in 2 These are trivial, but with null move, that can take 7 ply to find it. >22 mate in 3 These are easy, but with null move, that can take 9 ply to find it. >15 mate in 4 Think of how deep these are with R=3 null move >11 mate in 5 Think of how deep these are with R=3 null move >Which means there are 64 mate in 5 or less. A mate in 5 is ten ply. Given 5 seconds, that's not bad. >While it's usefull for a human to solve the same type of position again and >again it doesn't make much sense for a program. Your powerhouses which solve >290+ deal mostly with the remaining 10 positions which means 3.3%. >If you use another test like bt2630 you deal with about 20 of 30 positions which >is more IMHO. BTW I doubt that not using null move or using null move with R=2 >or R=3 will give much difference when running for very short times like your 5 >sec. Try it. >So I think the wac test as a test for programs may have been usefull in the >beginning of chess programming - but now it is obsolete. It is the only debugged test suite of any substance, and (as such) it is the most useful test suite for any purpose. There are other short test suites which have harder problems. But with such a small set, you might actually miss a lot more tactical shots by fixing your program to work better with them. I have a different suggestion: Create a huge "Did I break something?" set with 500 or so problems with an easy known solution that should be solved quickly. Create a large "These are tough problems but we know for sure that the answer is right" which might be useful for finer improvements. Create a set of problems which have never been solved. If you start solving them, then you are on to something. Nobody has ever bothered to do this. The short test suites are not useful by themselves. You must have a larger data set to be sure you have not broken something.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.