Author: Tord Romstad
Date: 13:44:04 04/12/04
Go up one level in this thread
On April 12, 2004 at 14:45:28, Christophe Theron wrote: >On April 12, 2004 at 07:50:47, Tord Romstad wrote: >> >>Assume that you make a change to your engine which improves the playing strength >>by >>about 10 Elo points. How many hours of CPU time do you need before you are sure >>that >>the change was an improvement? >> > >I would say approximately one week, and I would not even be really sure it is an >improvement. We are talking about a 1.5% improvement in winning percentage here, >it's below the statistical noise of a several hundreds games match if you want >95% reliability! Thanks, Christophe! Reading this is actually a great relief to me. I wondered if you had invented some kind of magic which enabled you to find tiny improvements in much shorter time. >And unfortunately a 10 elo points improvement is becoming rare for me. Most of >the changes I try make the program weaker, and many changes do not provide any >measurable improvement! I have no difficulties believing this. My engine is still at least 200 points weaker than yours, and I have exactly the same experience. >That's why not having a strong test methodology is totally out of question if >you are serious about chess programming. Yes. It is extremely difficult to me, because I am a very impatient person. When I make a small change to my engine, I rarely have enough time to play enough games to determine whether it is an improvement, because I have a dozen new ideas I want to try before my first test matches are finished. >Even with a good test methodology chess programming is still an art: in many >cases you have to decide with your feelings, because the raw data does not give >you a definite answer. > >Now of course there are small improvements that I do not even need to test for a >long time: if I find a way to make my program 10% faster without changing the >shape of the tree, then all I need to do is run some safety tests that will only >look at the number of nodes searched on a large set of positions and compare it >to the last stable version. Yes. That is the only good reason I see to do any low-level optimization (for weak amateur engines like mine, that is -- the situation is of course entirely different for authours of top engines). If I manage to make my engine a few percent faster without changing the eval or the shape of the tree, I can be 100% sure that the change was an improvement, without doing lots of time-consuming tests. Tord
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.