Author: Walter Koroljow
Date: 15:36:22 02/04/01
Go up one level in this thread
Bruce, I am glad you are willing to do something constructive. I am coming into this thread at this point, so forgive me if I do not fully understand the context. As I understand it, you want to measure things that are difficult to measure -- e.g., whether your latest small change added a rating point to your program. You also want to minimize testing time. To me this seems crucial for a chess programmer. This seems to be a case for sequential testing. I have an old "Handbook of Probability and Statistics" open in front of me. Let me give you two quotes from the sequential testing section. "Sequential tests are frequently more economical than non-sequential tests, especially if the number of trials is readily changed... Oftentimes the number of trials called for in a sequential test of a given reliability is considerably less than the number of trials required by the corresponding non-sequential test." "In a sequential test, certain calculations are made after each trial...the hypothesis is accepted or rejected as soon as it appears that the available data are adequate for making a decision..." This is probably what you are doing already, but the theory will give you a quantitative criterion for stopping testing. You might consider looking at the subject in a reference book or textbook, and seeing if you are comfortable with it. Incidentally, I would be glad to help with the analysis (if you wanted help), time permitting. I have a Ph.D in mathematical physics, the math courses for a Masters, and have been producing systems that measure hard to measure things for a living for 30 years. For example, my system (all design and analyses done by me) is the only system that can, and routinely does, measure the (miniscule) noise of the Seawolf submarine. So, I have done a few dozen statistical studies professionally. Anyway, best of luck. Walter
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.