Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Testing Chess Programs

Author: Christophe Theron

Date: 23:48:34 04/13/04

Go up one level in this thread


On April 13, 2004 at 15:09:10, Tord Romstad wrote:

>On April 13, 2004 at 14:21:07, Christophe Theron wrote:
>
>>You know, one has to wonder where the difference in elo strength between Crafty
>>and the top commercial comes from.
>
>My guess is that the following three factors (in order of importance) are
>the main reasons:
>
>1. No selectivity at all except for recursive null move and some futility
>   pruning.
>
>2. Very simplistic qsearch.
>
>3. Lack of endgame knowledge.



4. lack of a serious testing methodology

No offense intended, I do not think that Bob claims to have such a methodology.

1, 2 and 4 are the important points.





>I don't think it is entirely fair to compare Crafty to the top commercial
>engines, though.



If I said "Crafty sucks", that would be unfair. But here I am simply comparing
not only the strength but also what's inside the program. I can do that because
I am in the privileged position to have written a chess program that is stronger
than Crafty AND to be able to have a look at what's inside Crafty. So I know
what Crafty is missing.




>Bob's goals are not quite the same as yours.  His primary
>goal is not to create an engine which plays well on computers which most
>people buy, but to write an engine which takes advantage of very fast
>multi-processor machines.  I think the gap between Crafty and the top
>commercial engines would have been much smaller if Bob had concentrated
>on good selective search rather than parallel search.



I am not assessing the value of Crafty as a project here.

I have taken Crafty as a point of reference because it is a state of the art,
open source, academical chess program. Very few non-commercial projects have
reached this level, and it's the only one that is open source, so I'm not the
only one taking it as a reference point.

I do believe that the strength of Crafty would not be the same if Bob had
different goals.

Several years ago I had a very heated discussion with Bob here on the subject of
Crafty's strength. I do not want to start this again. I know the goals and value
of this project are somewhere else.

This probably explains point 4 above. There is no real effort spent on a serious
testing methodology because it's not the goal of the project to achieve more
strength by fine tuning all the parameters. That would probably make it less
easy to understand, and less easy to add new features.




>>One thing I am convinced of is that if the top chess programmers started to
>>exchange ideas, like Ed and I did, you would see a significant increase in the
>>strength of these top programs. Clearly some of them would benefit more.
>
>You're probably right.  I often wonder why other commercial programmers
>don't seem to make similar alliances (perhaps they do, but keep it secret?).
>
>Tord



When you think about it, some factors could favor such alliances, especially for
programmers working in the same company, especially if one is young and the
other one is older, preparing to leave the field. That is called commercial
pressure. But I do not know if it has already happened.




    Christophe



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.