Author: Christophe Theron
Date: 08:53:15 10/31/03
Go up one level in this thread
On October 31, 2003 at 00:03:05, Mike S. wrote:
>On October 30, 2003 at 22:21:39, Christophe Theron wrote:
>
>>(...)
>
>>Instead of giving CM9000 TWO unfair advantages it should:
>>* either not be tested at all
>
>Would you call it FAIR when a major competitor of yours wouldn't be tested at
>all??
>
>>* or be tested without opening (and without book learning of course)
>
>Would you call it FAIR to test a major competitor without opening book when
>others can use books??
Yes.
If the author/manufacturer of the engine does not make the necessary effort to
provide a book, he gets what he deserves.
>Are you demanding to replace something what you call unfair advantages, by
>unfair DISADVANTAGES?
Yes.
There's a real world out there, you know.
If you don't make any effort, you get slaughtered in the real world.
> Is this your serious opinion? You demand that a competitor
>of yours should either not be tested at all, or without book? Excuse me, but if
>this is really your opinion, I'm disappointed. Then, I would have to believe
>that your opinion is influenced too much by your personal interest.
The King is a very strong engine, maybe stronger than Chess Tiger.
I have the greatest respect for Johan and his work.
Are you happy now?
I'm just saying that no special favor should be given to any competitor.
When I asked the SSDF if it was possible to send an update to the Tiger 14
opening book, they required that the improved opening had to be made available
for free download to all customers immediately.
And it was a perfectly fair requirement.
So we made the improved book immediately available for free download to our
customers.
Johan has, I think, enough fingers to build a suitable opening book for the SSDF
test, to send it to the SSDF, and to make it available for free download.
I don't understand exactly why he would be exempted from this.
I don't understand why he would receive an unfair advantage as a reward for NOT
complying to the most basic rules.
>The conditions SSDF uses reflect *typical King 3.23 usage* among computer
>chessfans. It doesn't matter so much, which book is used. The general.ctg is
>neiter the strongest nor the newest book. I can imagine that is unpleasant to
>have to expect, that another engine joins the competition for SSDF rankings, but
>that shouldn't affect your opinion about fair testing as long as you want to try
>to be fair.
It is not unpleasant. I'm not #1 in the list. I'm not even #2 or #3 anymore.
What do I have to lose? Can you tell me? Dropping from #5 to #6? Oooh I'm
scared... :)
My opinion is that The King (CM9000) deserves to be in the list.
The problem is that allowing exceptions to the simple rules of the SSDF is going
to turn the list into pure garbage.
I'm in favor of fair testing. I have always been.
However this requires a minimal amount of cooperation from the engine's author.
Actually I believe that the author does not want to be in the list. Or does not
care. So why should the SSDF reward him by giving his engine TWO UNFAIR
ADVANTAGES?
>@Sarah, testing King defaults makes no sense! I'm surprised that somebody
>experienced in computerchess like Chessfun would really prefer to test King with
>CM defaults, when you must know that several custom settings are more than 100
>elo stronger. Nobody who knows the least about King uses the CM defaults, since
>CM8000. I think you know the Utzinger tournaments and results, the Grailmaster
>CM personalities ratings, etc.
You are right. Johan said it prefers the improved settings to be used, and
that's perfectly fine.
It has always been allowed, and it is a good thing. No problem here.
People want to know the best they can get of a single PRODUCT. That's the
question the SSDF ultimately wants to answer: "what is the best chess PRODUCT?".
Historically, the SSDF does not try to answer the question: "What is the best
product out-of-the-box?".
So applying better settings is allowed. No problem.
And it does not try to answer the question: "What is the best cyborg one can
build by using parts of all the other products around?".
So the recipe for a good SSDF test is:
* get a product
* apply the settings recommended by the author of the engine
* apply the various free patches known to improve the product
* test
Adding a new step "mix the product with whatever good thing you can find in
another product" is like opening the pandora's box.
>At what age children learn already, that almost *any rule has it's exceptions*.
>So please people, especially the regulars & experts, be a bit more realistic
>when discussing this. The SSDF decision is ok! An important strong engine will
>join the list (finally, and a bit late), tested in a reasonable setup. I would
>have recommended to generate a medium size "neutral" book from a good game
>collection for King in Fritz, but I guess it isn't so much difference from the
>general.ctg which SSDF has used before in similar cases (AFAIK Crafty and
>others).
>
>Btw. in a recent test Fritz 8 vs. Shredder 7.04, more than 250 games each,
>Shredder 7.04 *without book* scored only ~1% less against Frit+Fritz book, than
>with it's own book! So there's really no need to panic no matter which book an
>engine uses. Books are overrated (said i.e. Shay Bushinsky too).
>
>http://www.miko42.de/DasDuell/duellindex.html
>
>When general.ctg was used by SSDF for some somewhat weaker freeware engine
>tests, I don't remember any "nervouseness" about it :-))
It was already unfair.
Christophe
>This is really much too obvious...
>
>Regards,
>Mike Scheidl
Sorry, but your comment is shortsighted.
Many people who have no commercial interest in the SSDF results share my opinion
on this issue.
And I think that you greatly overestimate what my interests in this can be. The
rating of CM9000 will have absolutely no impact on my business.
Christophe
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.