Author: Sune Fischer
Date: 01:13:20 06/02/04
Go up one level in this thread
On June 01, 2004 at 20:08:37, Robert Hyatt wrote: >On June 01, 2004 at 19:25:41, Sune Fischer wrote: > >>On June 01, 2004 at 18:39:46, Robert Hyatt wrote: >> >>>>We wanted to know what the strength relations were with learning off, now we >>>>know. >>> >>>Why? >> >>Mainly because you want reproducability, it's no good to have an engine that's >>not performing at a constant level. >> >>Ie. suppose I play a match against Crafty, then I change something in my engine >>and wants to see if it got better. >>If Crafty learns, my engine will probably do worse even if it is an improvement. >> >>Surely you can see why that is a nonsese experiment to do, learning _must_ be >>switched off or I simply cannot test against Crafty. > >There is a solution. Clear the learning before you start a test. This won't work if I run tests of different lengths, I can't compare a 200 game match to a 100 game match. > But even >then, you have _real_ problem because there is some randomness built into my >move selection logic to provide variety. That's annoying yes, but as long as it averages the same strength it might not be totally damaging. >If you play a 20 game match, make >changes, and play another 20 game match, comparing the results is less than >worthless... So maybe Crafty is just worthless for testing, that is possible. >You can easily answer that. But you also wouldn't publish a result of such a >match without clearly identifying that Crafty was badly handicapped. That was >the point. If someone reads "book learning disabled" and they don't know the >whys and whatfors about book learning, they might say "so what, no big deal" >when it really is. Crafty has been around so long, people (that matters) knowns pretty well the level it's playing at. Give us some credit here :) >My philosophy has _always_ been one of "don't whine about a problem, fix it." Nothing wrong with that of course, but why complain if some decides to disable the cause of all the problems and thus remove the problem itself? I think I see where you are comming from though. Because you've fixed it the problem should always hang around, so that everyone else is doomed to spend an equal amount of time in fix it too, or else it's not "fair"? >Several years ago Ed was complaining about "duplicate" games in the SSDF testing >that was being done. I thought about that and decided "rather than complaining >about duplicate losses, I'm going to simply avoid them by having crafty notice >that it got into trouble in an opening and not play it again for a while." That >is where my "book learning idea" was founded. A problem that you could either >complain about (does it make sense to let a program lose the same opening over >and over and count that against it and for its opponent?) or solve. I chose >"solve" and have not had the problem happen to me, at all... > >Of course if you turn it off, the problem comes right back, bigger than life, >and sticks around. Of course Ed has no control over how SSDF does their testing, if he cares about their result he must "do it their way". -S.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.