Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Testing Chess Programs

Author: Robert Hyatt

Date: 10:36:08 04/13/04

Go up one level in this thread


On April 12, 2004 at 23:07:46, Christophe Theron wrote:

>On April 12, 2004 at 19:51:29, Tom Likens wrote:
>
>>On April 12, 2004 at 19:15:01, Christophe Theron wrote:
>>
>>>On April 12, 2004 at 16:44:04, Tord Romstad wrote:
>>>
>>>>On April 12, 2004 at 14:45:28, Christophe Theron wrote:
>>>>
>>>>>On April 12, 2004 at 07:50:47, Tord Romstad wrote:
>>>>>>
>>>>>>Assume that you make a change to your engine which improves the playing strength
>>>>>>by
>>>>>>about 10 Elo points.  How many hours of CPU time do you need before you are sure
>>>>>>that
>>>>>>the change was an improvement?
>>>>>>
>>>>>
>>>>>I would say approximately one week, and I would not even be really sure it is an
>>>>>improvement. We are talking about a 1.5% improvement in winning percentage here,
>>>>>it's below the statistical noise of a several hundreds games match if you want
>>>>>95% reliability!
>>>>
>>>>Thanks, Christophe!
>>>>
>>>>Reading this is actually a great relief to me.  I wondered if you had invented
>>>>some kind
>>>>of magic which enabled you to find tiny improvements in much shorter time.
>>>>
>>>>>And unfortunately a 10 elo points improvement is becoming rare for me. Most of
>>>>>the changes I try make the program weaker, and many changes do not provide any
>>>>>measurable improvement!
>>>>
>>>>I have no difficulties believing this.  My engine is still at least 200 points
>>>>weaker than
>>>>yours, and I have exactly the same experience.
>>>>
>>>>>That's why not having a strong test methodology is totally out of question if
>>>>>you are serious about chess programming.
>>>>
>>>>Yes.  It is extremely difficult to me, because I am a very impatient person.
>>>>When I make
>>>>a small change to my engine, I rarely have enough time to play enough games to
>>>>determine
>>>>whether it is an improvement, because I have a dozen new ideas I want to try
>>>>before my
>>>>first test matches are finished.
>>>
>>>
>>>
>>>That's the real motivation killer. I also have many ideas and when I want to try
>>>them I realize I'm currently testing another idea and that the test running will
>>>not be over until next week. So I have to wait for one week before I can start
>>>testing, and another week to know the result.
>>>
>>>In two weeks from now, my interests will clearly have switched to another idea.
>>>
>>>That makes computer chess programming more and more boring.
>>>
>>>
>>>
>>>    Christophe
>>
>>I was thinking about this *exact* problem on the way home from work today.
>>The only solution I could come up with was to add more computers and thus
>>attack the problem in parallel.  I currently have three computers I can
>>dedicate to running various test matches, if I could validate an idea in
>>roughly two days then this problem wouldn't be so bad.  A week, as both you
>>and Tord point out, is difficult.  Adding more CPUs to the problem would
>>make this possible, but it might also turn me into a bachelor again!!
>>
>>As I mentioned previously, testing is my primary focus for the next few weeks.
>>If I come up with anything interesting I'll share it (not being commercial
>>does have a few advantages).  Also don't hesitate to *not* share anything
>>since you make your living at this I can appreciate your position.  If
>>someone asked me to design an integrated circuit for free, I might be
>>reluctant to do so (especially, if it resulted in my not being able to design
>>one for a paycheck in the future).
>>
>>regards,
>>--tom
>
>
>
>I consider that I *do* contribute to computer chess programming. Not by
>providing code, but by providing advices.
>
>I'm not doing what Bob does. Bob provides excellent advices on code, or code
>structure.

I'm not even sure that "what Bob does" is correct.  IE based on all the "clone"
problems it has caused, sometimes it seems that the cure has been worse than the
disease..  :(


>
>I provide more general, or philosophical, advices. They do not cover the same
>areas as advices provided by other people. Some of them took me years to come up
>with, so from my point of view they are valuable, maybe more than code.
>
>On the other hand I'm still learning myself, so sometimes my advices are not
>that clever. :)
>
>Further, wouldn't you just *hate* if I took the fun out of chess programming by
>telling you everything? :)
>
>
>
>    Christophe
>
>
>
>
>
>>>>>Even with a good test methodology chess programming is still an art: in many
>>>>>cases you have to decide with your feelings, because the raw data does not give
>>>>>you a definite answer.
>>>>>
>>>>>Now of course there are small improvements that I do not even need to test for a
>>>>>long time: if I find a way to make my program 10% faster without changing the
>>>>>shape of the tree, then all I need to do is run some safety tests that will only
>>>>>look at the number of nodes searched on a large set of positions and compare it
>>>>>to the last stable version.
>>>>
>>>>Yes.  That is the only good reason I see to do any low-level optimization (for
>>>>weak
>>>>amateur engines like mine, that is -- the situation is of course entirely
>>>>different for
>>>>authours of top engines).  If I manage to make my engine a few percent faster
>>>>without
>>>>changing the eval or the shape of the tree, I can be 100% sure that the change
>>>>was an
>>>>improvement, without doing lots of time-consuming tests.
>>>>
>>>>Tord



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.