Author: Enrique Irazoqui
Date: 02:55:52 04/28/98
Go up one level in this thread
On April 28, 1998 at 04:00:31, Ed Schröder wrote:
>Hi Thoralf,
>
>Thanks for your comments, I like to make my position clear.
>
>First, I have been in love with your list and your work since SSDF was
>founded. What year was that? I believe it was 1984. In that 14 years
>you have build yourself quite a reputation as the main (almost only)
>source for testing commercial and other general available chess software
>concerning playing strength by playing thousands of computer-computer
>games.
>
>You earned a lot of credit, like to quote my own home page now:
>
> SSDF is founded in 1984 and has been since then the general
> accepted computer rating list for chess magazines and computer
> chess lovers all over the world. The SSDF received this credit
> by their excellent work concerning honest an decent testing
> methods.
>
>In the past I have criticized your list, last time I did that was when
>Rebel 8.0 was on top of your list leading with 60 points. That was 1.5
>year ago in RGCC, remember?
>
>As far as I am concerned criticism is meant and should be used to
>improve
>the quality of your work. It's then up to you to decide what to do with
>the information.
>
>A few months ago it became known that Rebel8 and Rebel9 were tested on
>SSDF with its main book turned off. That was a huge mistake. I was not
>very happy with this, still I saw no reason to have any doubts on the
>creditability of your list. I understand these things sometimes happen.
>
>But what is happening right now in unacceptable for me. I like to
>compete on your list (still do) and have competed on your list since 12
>years starting with Rebel 5.0 (5 Mhz). Here are my points....
>
>#1. It's my opinion that every chess program that competes on your list
> should be general available (see your own remark above)
The engine, opening books and table bases of Fritz 5 as tested by the
SSDF are generally available. The autoplayer is not.
>#2. You made a first mistake on #1 (your own rule) when you accepted
> to test Genius4 as a DOS version while the commercial program was
> a Windows version. This DOS version of Genius4 was not available
> for the public.
>
> This information became publicly known after the Genius4 program
> was on your list for a long time. I remember from RGCC you received
> criticism for that.
>
> You have given the program an unfair advantage above other
> competitors on your list mainly because the DOS version could now
> use much larger hash tables. We are talking about the days of 486
> and P90 machines with 4 or 8 Mb memory.
>
>#3. Now we have the Fritz5 case and it shows you haven't learned from
> the Genius4 DOS mistake. You have ignored the criticism (and the
> principals behind it) you received (mainly from RGCC) on the Genius4
> DOS case where you sinned against your own rules.
>
> In fact you have made the problem bigger. You accepted to
>participate
> a program that DEMANDS a 64 Mb machine so that it can use a big hash
> table which gives the program an unfair advantage above other
>programs
> who have to play on 32 or 16 Mb machines. This is unfair
>competition,
> you may correct me if I am wrong.
I also think programs should be tested on identical platforms.
> Then I hear about the use of power books, endgame Cdroms which are
> not part of the commercial package of the program. Minor points IMO
> concerning the gain of some extra elo points. But another proof of
> the tendency that you sin against your own rules.
>
Endgame table bases are part of Fritz5's Cdrom. Powerbooks are
commercially available.
>#4. But my main criticism is that you have accepted an unknown piece of
> software for autoplaying on your list. The autoplayer behaves
>totally
> different then the general accepted auto232 software. And then the
> worst came, the new autoplayer is secret. Again you sin against
> your own rules.
I think this has been the major issue.
1 - The autoplayer in Fritz 5 being suspect of "cheating", as some
suggested. You and I proved this is not the case and the autoplayer in
Fritz 5 is clean.
2 - By not being available to programmers, ChessBase can test other
programs and tune Fritz against them and their books, while other
programmers can not test Fritz. In my opinion, this is not fair. I see
two ways out:
a - ChessBase release the autoplayer Fritz 5.
b - The other programmers release an autoplayer version of their
programs only to the SSDF and other testers.
I think a) is by far the best solution. With b) we solve the problem
with the SSDF, but other people interested in autoplaying will be let
down.
>#5. You know very well that auto232 is a fragile piece of software and
> easily can be miss-used still you accepted that other competitors
> or other participants on your list were simply sent in the dessert
> as nobody could check this piece of software.
The practical consequences of this point have been cleared by proving
the autoplayer Fritz 5 is fair. Proven after the fact, I know, but
proven.
>#6. As a result of that (and this is not funny at all) I made a special
> Rebel auto232 version which checks every auto232 opponent for fair
> play. This version was sent to somebody who has the secret Fritz5
> autoplayer. This special Rebel version reported, all ok.
Confirmed.
>Now for the future of your list I recommend you the following
>improvements
>which is a simple request to return to your basics and your own
>rules....
>
>#1. Only test programs as they come in the commercial package.
>
>#2. Also test freeware programs like Comet, Crafty, Decade as they are
> distributed. All these programs have in common that they are
> generally available to the public. Nothing should be hidden.
>
>#3. Do not accept any add-on's like separate books, endgame cdroms if
> they don't come with the product.
Unless they are commercially available too.
>#4. Do not accept autoplay software that is not public available. Stick
> to the standard auto232 protocol all programmers have agreed on.
Unless, see above, you do the same and send to the SSDF and to no one
else your own autoplaying Rebel. Less than ideal, but fair enough.
>#5. Test programs on equal hardware. Do not accept special demands which
> gives a program an advantage above others.
Absolutely.
>#6. Test with equal hash table sizes as far as this is possible. Do not
> accept demands for a minimum hash table size other then the system
> requirement that comes on the box of the program. Everything else is
> a clear try to get an unfair advantage which you should not accept.
>
>If you return to these rules again which were common in the past it will
>be my pleasure to compete on your list again.
I truly hope so. In my opinion, all this issue about Fritz 5 and the
SSDF has been blown out of proportion. I understand very well your
criticism, but I do believe there are feasible ways out.
Enrique
>- Ed Schroder -
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.