Author: Don Dailey
Date: 11:26:30 12/13/97
Go up one level in this thread
On December 13, 1997 at 13:28:10, Thorsten Czub wrote: >>Some people think that they can tell which of two programs is stronger, >>by eyeball, but I'm mistrustful of this. If one program beats another >>7-3 (a statistically insignificant result), which one do you think they >>will pick? How often do you see computer vs computer losses in which >>the loser looks good? > >But isn't it stupid to refer to a RESULT when you have the GAMES ?? > >If you have seen 3 games, than a result is much more accurate. Statistic >is ok IF YOU HAVE NOT MORE DATA THAN THE SCORES, but I would always >prefer having seen the games myself instead of getting 5 or 7 times >more game results. >If I watch 2 computers play, I can say very early which program is >better and why. >I have no evidence. But a feeling. AND: you can proof this feeling by >waiting 3 months until the autoplayer has outplayed some reasonabnle >data. >So why mistrust feelings when I can proof them ???? You can prove nothing here I'm afraid. You can claim one program is better based on the games and then with a lot of EMPIRICAL evidence the evidence can be said to be in your favor at best. I could also take a completely random guess and have a 50/50 chance of being "proven" correct. But the only thing it would prove is that I'm a "lucky" guesser! If I consistantly make the right choice then there would be some evidence that I knew what I was talking about. > >Of course you can say YOU mistrust other people's feelings, but why >should I mistrust MY feelings when I have not made many mistakes over >the years ?? I gave an example earlier of how strong players believed one program was better than another by looking at the games, but they were wrong. Let's say we get a 7 to 3 score but in fact the it's the stronger player that lost this very short match. Now you are telling me that you could look at the games and see that in fact the guy who scored 3 points is the stronger player? But the games themselves indicated that the weaker player in fact played the better game (for whatever reason.) I don't want to contradict you but I have serious doubts that you or anyone is this good a judge. Of course if you were show a hundred games you would notice trends, strengths and weaknesses of each program but I'll bet you would not cast your vote in favor of the lower scoring player. Here is a thought experiment: Take 2 programs, A and B such that A is 100 rating points stronger than B based on lots of evidence. Play a series of games. Pick out 10 wins from player A, 5 white and 5 black. Do the same for player B. Make sure the games are selected completely at random. Do you believe you can easily tell who the better player is? I am betting that you have no better than a 50 percent chance of correctly identifying the better player. In my own testing, I've noticed that often, when we win against another strong opponent, we usually look like clearly the best player. And when we lose the opponent looks like the better player. I mean, give me a break, if you play the better game you will not lose the game, even if you are the weaker player. Now it's possible, that if you look at a lot of games based on my thought experiment you might be able to pick up a trend eventually, perhpas the wins seem to come easier for one side than the other. But again, I say this will require a very large sample size to detect. If the difference in strength is huge, you can probably see this with a smaller sample, which is exactly the way it is with normal testing. I think it's unlikely that you are blessed with the ability to pick out the better player QUICKLY from just looking at the games, except by noticing that the better player keeps getting better positions (and does not screw them up) i.e. he is winning more games! -- Don
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.