Author: Thorsten Czub
Date: 11:07:42 09/06/98
Go up one level in this thread
On September 06, 1998 at 07:08:11, Christophe Theron wrote: >>Also i would never trust games version x against x+1 because the 2 engines are >>too similar to get real data to trust in. > >You repeat this at will, but never give good reasons. I do always give good reasons ! :-) My experience over the years with version x against version x+1 has shown me, that this method is not reliable. This has to do with the similarity of the engines. The 2 engines fighting against each other having nearly the same evals, but version x+1 has some extra stuff, makes x+1 normally superior. But with using this x vs. x+1 you don't know how this environment behaves in games against all other programs or (even more complicate to find out) against all kind of opponents (humans/computers). You only proofed that the new changes of version x+1 make it "stronger" relative to x, but your experiment has shown no data to abstract this "stronger" term on all the other circumstances, as: playing against an opponent NOT = x but y. You get fooled by the incest problem. When I do a tournament, and ssdf did this in the past, with playing richard lang x (16bit) , richard lang x+1 (16bit) , richard lang x (32bit) , richard lang x+1 (32bit) , and some other programs, than you get statistics that avoke wrong data. Because version x+1 will maybe be stronger than version x, and normally version 32bit should be stronger than version 16bit, but this does not tell you anything if x is stronger than y against others. I have had programmers doing the x vs. x+1 method over the years. Especially in the old days. And whenever he sent me a version and said: X+1 got 70% against X then there was NO guaranty that this version was REALLY stronger in general. This is my experience. Not any proof. Only my experience. That i had often the problem that these 70% are not REAL, they are caused by the fact that x is x+1 alike... >I know several aspects of computer chess where the only way to check if a change >is good is doing self test. SELF test ?! I don't think SELF test (x vs. x+1) makes much sense. Sorry. My method is different. I watch a game, or more games, and try to find out subjective way, using my feelings and emotions. And main-lines. And i do not bean count nor use statistics. > > Christophe We don't have to agree here Christophe. This world is open for any opinion.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.