Author: Dan Homan
Date: 13:21:23 06/16/99
Go up one level in this thread
On June 16, 1999 at 14:52:47, Melvin S. Schwartz wrote: > >On June 16, 1999 at 13:28:24, Dan Homan wrote: > >>On June 15, 1999 at 23:47:07, Melvin S. Schwartz wrote: >> >>> >>>I disagree. They're running programs on different hardware and that doesn't make >>>for intelligent evaluations of program vs. program. Furthermore, I didn't say >>>they shouldn't do it, but rather what is to be accomplished by testing program >>>against program on various types of hardware that is not of equal stature. They >>>can do it - but is it truly meaningful??? >>> >> >>Depends on what you mean by meaningful. This contest is to find the >>best artifical chess player. I think that is pretty meaningful. >> >>Notice that I said "player" not program. Clever algorithms are only >>one component of a chess player. Hardware is important too. Some >>artifical players use special purpose hardware.... Deep Blue for >>example. The question is: "What is the best artifical player?" >> >>Now, if you want to use the results to say something about the >>relative strength of the algorithms you can buy for your home >>computer, you are out of luck.... The results from this contest are >>not meaningful in that particular way, but they are meaningful >>in other ways. >> >>If you still are doubtful, we could turn this around. Suppose that >>you have organized a tournament. In your tournament all the same >>kinds of computers are used and all the newest commercial software >>is playing. Now, I could critize your tournament as not being >>meaningful because it doesn't tell us what the best "artificial >>chess player" is. By not including other kinds of artificial chess >>players and other types of hardware, I could say that your results >>were tainted. >> >>If I said these things about your hypothetical tournament, I would be >>dead wrong because I would be putting my meaning into your results >>rather than looking at what you were trying to do. Your results would >>tell us which commercially available program is best on the hardware >>you selected. > >Hello Dan, > >If the programs were running on the same type of hardware, I believe that would >yield results which could be intelligently evaluated. If you run program A at >600 MHz and program B at 200 MHz, what possible intellectual conlusion could you >come to if program A defeated program B? That program A at 600 MHz is likely to be better than program B at 200 MHz. It is the software+hardware that is being compared here, not the software alone. When someone critizes the methods used in an experiment/test/competition it is usually for one of two reasons: 1) They have discovered some genuine flaw in the methodology. or 2) They have mis-understood the question to be answered by the experiment/test/competition. Your case is #2. I am a scientist and have this kind of mis-understanding myself all the time, especially when I am reading about a new experiment. It is important to get such mis-understandings cleared up. You confusion comes from not understanding the question that is being asked by this tournament. Please read my previous post again. Here is a example which doesn't involve chess. Suppose I create a race to find the fastest wheeled vehicle on the planet. People can bring whatever they like, as long as it travels on wheels. Some will bring jet-engine powered monsters. You might critize this race by saying that allowing anything on wheels to compete will prevent you from using the results to decide which car to buy, but such a critism would miss the point of the competition. Back to chess: Once you understand that the point of the WCCC is to find the best artifical chess player in the world (no restrictions) and not to compare program A to program B on equal hardware, you might choose to think that finding the best artifical chess player is silly. That is fine because you will have understood the question asked by the competition. - Dan > >Mel >> >> - Dan >> >> >>>Mel >>> >>>>TP
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.