Author: Slater Wold
Date: 16:47:53 09/12/01
Go up one level in this thread
On September 12, 2001 at 19:28:15, Dann Corbit wrote: >On September 12, 2001 at 19:13:13, Bruce Moreland wrote: > >>On September 12, 2001 at 18:38:07, Slater Wold wrote: >> >>>I tested Crafty against the 300 WAC positions, yesterday and today. >>> >>>The tests were run with 48M of hash, and 10M of hashp. Also all 5 piece TB were >>>in place, and the machine was a 2x1.4Ghz that gets roughly 1.41M nps on the >>>bench test. (That's with the hash. With nothing but smpmt 2, it gets 1.44M.) >>> >>>1 second = 275/300 (Interestly, Deep Shredder get 274/300 in this test.) >>> >>>3 seconds = 293/300 >>> >>>5 seconds = 294/300 >>> >>>10 seconds = 295/300 >>> >>>60 seconds = 296/300 >> >>WAC is a dumb test when you have a program at this level. You can throw out all >>but about 10 of them, and you still have something that's hard to use to test a >>program at a sensible time control, because the remainder are either too easy >>(still) or too hard. > >I think you're wrong. > >In fact, I think that this test may have uncovered some problem with crafty. >I used to get a better score on a slower machine. I suspect some minor thing >may have got broken. Either than, or an evaluation term may have been changed >that plays better in real games but plays worse on test suites. > >I don't think you can throw out any of them, unless it gets them right. How >will you know it got them right unless you run it. > >As a side point, sometimes chess engines discover new solutions to the problems >when run at longer time controls. That is an interesting finding, to me at >least. > >What are you trying to accomplish when you run a test suite? I believe that test suites are a good "benchmark", more than a test. I believe that when tuning a chess program, a test suite is just as effect as a quarter mile is to a race car. You tune, you gain a second. You tune, you lose a second. As long as the positions are "real-world" positions and not something that never happens in a real game. (Like leonid's mates.) WAC positions are from real world chess games. And the benefit of them, is there are so many of them. So therefore (refering back to the race car) say your "top end" or "endgame" improves, while your "60 foot time" or "opening" suffers. It's give and take. Slate
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.