Author: Sune Fischer
Date: 09:16:16 06/02/04
Go up one level in this thread
On June 02, 2004 at 11:42:24, Robert Hyatt wrote: > >> >>> But even >>>then, you have _real_ problem because there is some randomness built into my >>>move selection logic to provide variety. >> >>That's annoying yes, but as long as it averages the same strength it might not >>be totally damaging. > > >Playing the Sicilian in one match as black, the Latvian in the next match will >not "average the same strength"... > Doesn't matter, what matters is playing the sicilian and Latvian at constant levels. Eg. suppose that in one of our basement tournaments Crafty gets the same sicilian twice against two engines of equal strength. Crafty falls into a trap in the first game, "learns" and manages to avoid it for the next match. This will only punish the latter of the two opponents. I think that is unacceptable in a testing environment. >>>If you play a 20 game match, make >>>changes, and play another 20 game match, comparing the results is less than >>>worthless... >> >>So maybe Crafty is just worthless for testing, that is possible. > >Or perhaps your testing methodology is worthless... Crafty is not that >different from any other program. You have to be sure to play enough games to >hide the random factor. I can't do that against Crafty, if it learns it's a moving target. Testing is already hard enough, you don't need to throw additional random factors into the equation. >BTW, you have greatly changed the original point of my post... I _clearly_ >asked "why learn=off in a tournament that was being played." I didn't ask "why >learn=off in a test match for a single program?" > >If a _programmer_ wants to test his program against some crippled version of >Crafty, that's one issue. It is _not_ the issue that was being discussed until >you twisted the conversation in that direction... I was _specifically_ talking >about someone playing a basement tournament or basement match, not someone >trying to develop an engine... Why should there be a difference? The poster just had an interest in seeing how the engines did under those conditions, nothing more nothing less. If Crafty is greatly handicapped by disabling learning then I think it's still interesting to find out how much weaker it is. >>>My philosophy has _always_ been one of "don't whine about a problem, fix it." >> >>Nothing wrong with that of course, but why complain if some decides to disable >>the cause of all the problems and thus remove the problem itself? > > >Because there _is_ no "problem" to remove. It actually _adds_ a problem, rather >than removing one. If there is no problem, why did you go through all that trouble? >>I think I see where you are comming from though. >>Because you've fixed it the problem should always hang around, so that everyone >>else is doomed to spend an equal amount of time in fix it too, or else it's not >>"fair"? > >That is one view, yes. If you don't ponder, do you turn it off in my engine? If it's a ponder on tournament I play with the handicap, if it's ponder off it gets disabled for everyone. Couldn't be simpler. >If you don't program endgame knowledge, do you adjudicate all games once they >reach the endgame? >You _must_ fill in the holes you have, because in real events you can't hide >them by pretending everybody has them and bypassing the issue in some artificial >way... I don't know what you mean by real events. Anything is real or unreal pending your perspective. I'm mostly interested in events that are well controlled and statisticly significant, anything else is just a bit of fun. Well okay perhaps great fun, but still nothing that can be taken too seriously. > > >My point _exactly_... "their way" == "only way". Says who? AFAIK they don't test in smp mode, so there are definitely other ways. -S.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.