Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: sliding attacks in three #define

Author: Omid David Tabibi

Date: 09:06:59 04/13/04

Go up one level in this thread


On April 13, 2004 at 11:33:06, Tom Likens wrote:

>On April 13, 2004 at 09:40:41, Omid David Tabibi wrote:
>
>>On April 12, 2004 at 14:45:28, Christophe Theron wrote:
>>
>>>On April 12, 2004 at 07:50:47, Tord Romstad wrote:
>>>
>>>>On April 12, 2004 at 00:09:48, Christophe Theron wrote:
>>>>
>>>>>On April 11, 2004 at 13:52:59, Tom Likens wrote:
>>>>>
>>>>>>On April 10, 2004 at 21:53:17, Christophe Theron wrote:
>>>>>>
>>>>>>>On April 10, 2004 at 15:55:17, Tom Likens wrote:
>>>>>>>>
>>>>>>>>I'm not sure where I come down on the bitboards vs. non-bitboard
>>>>>>>>architectures.  My engine is a bitboard engine, but that doesn't
>>>>>>>>necessarily mean that the next one will be bitboard based.
>>>>>>>>
>>>>>>>>I don't believe though, that because no bitboarder has topped the
>>>>>>>>SSDF list that this really constitutes any kind of proof.  My strong
>>>>>>>>suspicion is that if all the top commercial programmers converted
>>>>>>>>over to bitboards tomorrow (yourself included) that *eventually*
>>>>>>>>their new engines would again rise to the top of the SSDF.  I'm
>>>>>>>>beginning to suspect that creating a strong (i.e. world-class) engine
>>>>>>>>involves a helluva lot more than just the basic data representation,
>>>>>>>>but instead involves...
>>>>>>>>
>>>>>>>>1. 24/7 dedication
>>>>>>>>2. A *real* way to measure progress
>>>>>>>>3. A selective search strategy that works 99.99999% of the time
>>>>>>>>4. Attention to about 2^64 minor details
>>>>>>>>5. A failed marriage (okay, maybe this is extreme but you see the point)
>>>>>>>>
>>>>>>>>regards,
>>>>>>>>--tom
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Number 5 (or something close) was the reason why Tiger has made such a progress
>>>>>>>between 1997 and 1999. :)
>>>>>>>
>>>>>>>Number 2, seriously, is worth spending several months on it.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    Christophe
>>>>>>
>>>>>>This has been my main focus over the past few weeks.  It's become readily
>>>>>>apparent to me that the improvement slope from here on up is much steeper
>>>>>>and I rather not waste my time implementing features that I can't properly
>>>>>>test.
>>>>>>
>>>>>>regards,
>>>>>>--tom
>>>>>
>>>>>
>>>>>
>>>>>That's the secret of real professional chess programmers.
>>>>
>>>>Of course you don't want to reveal your secrets, but it would be interesting if
>>>>you could answer
>>>>the following question:
>>>>
>>>>Assume that you make a change to your engine which improves the playing strength
>>>>by
>>>>about 10 Elo points.  How many hours of CPU time do you need before you are sure
>>>>that
>>>>the change was an improvement?
>>>>
>>>>Tord
>>>
>>>
>>>
>>>I would say approximately one week, and I would not even be really sure it is an
>>>improvement. We are talking about a 1.5% improvement in winning percentage here,
>>>it's below the statistical noise of a several hundreds games match if you want
>>>95% reliability!
>>>
>>>And unfortunately a 10 elo points improvement is becoming rare for me. Most of
>>>the changes I try make the program weaker, and many changes do not provide any
>>>measurable improvement!
>>>
>>>That's why not having a strong test methodology is totally out of question if
>>>you are serious about chess programming.
>>
>>Devoting a whole week with 5 computers working 24/7 is a luxury few can afford.
>>During the past two years I have developed Falcon from a 2200 engine to a 2700+
>>engine it currently is, all on one humble P3 733MHZ machine.
>
>Hello Omid,
>
>This may well true, but you can't deny that the extra CPUs would
>have been nice.

Of course, they will be most helpful.

>Also you *must* have a systematic way of testing
>changes or I don't see how you can make real progress.  It's too
>easy to make "obvious" changes that ultimately weaken the engine.

My usual testing is just running a long series of guantlet matches against
Shredder, Junior, Fritz, Hiarcs, Tiger, Deep Sjeng, Ruffian, Crafty, and a few
other engines.


>
>>In order to reach a 2700 level, the search should already be good enough.
>>But beyond that level, it is mostly the evaluation that matters.
>
>I don't disagree with this, but for the engines at the level of Djinn
>search is paramount.  Adding an extra ply of search depth is worth a
>huge increase in playing strength.  Interestingly, these results have
>even held true for stronger engines (2600+).  Adding extra search
>depth increases their playing strength as well- the leveling off of
>playing strength (i.e. the diminishing returns) everyone expected hasn't
>really happened (or at least not in the articles I've read).

As your engine becomes stronger, it is harder to find new ways of improving the
search considerably. But you will always have an endless "to do" list of
evaluation improvements.


>
>>Since the Graz WCCC, I have been spending almost all my time working on
>>evaluation function. The work on search has been limited to modifying one
>>pruning here, one extension there, etc. But again, beyond 2700, it is
>>evaluation that matters. And I fully agree with Vincent on that.
>>
>>It is almost impossible to test a single evaluation change to see whether it
>>improved the strength. If you change the evaluation of knight outposts by a few
>>centipawns, good luck testing it... In those cases you have to highly rely on
>>your feelings and chess knowledge, and then after doing many changes, test them
>>as a whole to see if they improved the strength. Just my two cents.
>
>This reminds me somewhat of the optimizing programmer who "knows"
>where his program is spending all of its time and therefore doesn't
>need to use a profiler.

For speed optimization, a good profiler is absolutely necessary. Another
recommendation: When optimizing the engine, don't change anything else
(evaluation, search, etc). Run the benchmark, write down the node count, and
after each major optimization run the benchmark to make sure the node count
remains the same. If not, you would immediately know that the last optimization
you made is to blame. This will save you a huge amount of work later.



>Since I'm not a strong chess player I can't
>rely on my intuition, so instead I need to be more scientific or I'll
>never make any progress.
>
>regards,
>--tom
>
>>>
>>>Even with a good test methodology chess programming is still an art: in many
>>>cases you have to decide with your feelings, because the raw data does not give
>>>you a definite answer.
>>>
>>>Now of course there are small improvements that I do not even need to test for a
>>>long time: if I find a way to make my program 10% faster without changing the
>>>shape of the tree, then all I need to do is run some safety tests that will only
>>>look at the number of nodes searched on a large set of positions and compare it
>>>to the last stable version.
>>>
>>>But that does not happen often! :)
>>>
>>>
>>>
>>>    Christophe



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.