Author: Christophe Theron
Date: 16:30:21 12/31/97
Go up one level in this thread
On December 28, 1997 at 19:37:36, Don Dailey wrote:
>Here is how I do SELF testing and some of my thoughts on it.
>
>I have 200 very shallow openings and they go to exactly 5 moves for
>each player or 10 ply. Larry Kaufman picked them so that they are
>all relatively normal, span a lot of theory but do not get you too
>deeply into the game (so we test opening heurstics too.)
>
>200 openings gives us a total of 400 games, so we try to test in
>batches of 400 games.
How long does a 400 games match takes to get accurate results?
>There are 2 types of levels I use, fixed depth and fixed nodes.
>Fixed depth is self explanatory and I rarely use this level but
>can be useful for debugging and proving the value of certain
>algorithms. For instance if I try out a new extension idea it
>better at least improve the play at fixed depths.
>
>The other level is more interesting. It is fixed nodes and I
>designed it specifically for self testing. The program will
>search until the exact number of nodes specified has been searched
>and then will stop. This is very nice for self testing because
>it gives me platform independence. I have a lot of systems I
>can, and do, test on and they vary greatly in the power of the
>harware and even the operating system. But I can directly
>compare any result from any platform without any worry when I
>do fixed node testing.
Yes. I use this kind of "fixed nodes" search to measure my program on
problems set. I use the number of nodes seen to solve a problem to get
an "offset" from an hypothetical Elo base (with a log correction). I
built that on the hypothesis that doubling the program's speed (or
dividing the number of positions seen to solve a problem) increases the
program's Elo by, say, 60.
Some variations are interesting when you use this rating method on
problems set. You can see the Elo variation on fixed depth (using a
problem set in which you don't care if the solution was found or not),
or by problem solving (stop the search as soon as the solution is found,
or stop the search at the end of the iteration).
I think this method is very interesting to detect tactical improvements,
so I apply it after a change in the selection or extension algorithms.
Your idea to use it in self-play is interesting also.
>When I test 2 identical versions I always get exactly a 50-50
>result. I designed my program to be completely deterministic
>(in these kind of games) and this has also proven to be a good
>debugging aid.
The "fixed nodes" method has this great advantage to give an exact 50%
score in testing a program against himself!
>I never do thinking on the opponents time with this kind of testing.
>I believe that it has no serious impact on the results (since both
>sides would benefit from it equally) and that because of this it is
>a waste of machine resources. Also it would completely defeat all
>the effort I went to to isolate the hardware performance from the
>results.
Yes, and your goal is not to test the efficiency of the "permanent
brain"!
>Sometimes when I'm at home I'll get a brilliant idea, implement it
>and then log in to several computers at work and start tests running.
>when I get to the lab the next day I will have hundreds of games ready
>for me. Usually my brilliant idea was not so brilliant!
>
>
>-- Don
Oh yes, it happens to me every day!
Thanks for giving us your methodology. It was very interesting!
Christophe
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.