Author: Christophe Theron
Date: 09:42:00 11/10/00
Go up one level in this thread
On November 10, 2000 at 09:03:52, José de Jesús García Ruvalcaba wrote:
>On November 09, 2000 at 12:47:42, Christophe Theron wrote:
>
>>
>>
>>The difference is that the SSDF plays LONG tournaments which are more reliable
>>than the short WMCCC for example.
>>
>
>Sorry but I do not see any proper tournament in SSDF, only long mathces (20 or
>40 games). There are several weak points in SSDF testing, a major one is that
>there is no recipe or formula to give opponents to the new entries (as in a
>tournament, be it short or long).
It's nothing compared to the errors introduced by the weak opponents you find in
the last WMCCC for example.
You can argue that the swiss pairing system is supposed to do its best under
these conditions, but anyway the conditions are so bad that the swiss system is
unable to balance this problem.
In the tournaments, the number of games is so low that any top program that has
the luck to play a very weak opponent gets a huge advantage.
Such unfairness does not exist in the SSDF, at least it exists but to a much
lesser extend.
This is still an argument in favor of the SSDF in my opinion.
> I do not insist on the these weak points
>because I do not mean to be discouraging or overly critic to SSDF, I only want
>to point that your "reliability" argument in favor of SSDF is not fully
>convincing.
The argument actually shows yet another weakness of tournaments.
>>So there is a difference, but it does not favor the short tournaments in my
>>opinion.
>>
>
>Again, I respect your opinion, and I think you will have far more "followers"
>than me.
>
>>
>>Then you should value more the SSDF games than the tournament games.
>>
>
>OK, more problems with SSDF:
>-several (most?) games are not public. If a tester is too lazy to send a pgn
>score, she/he may be too lazy to even play the games.
Yeah, sure.
One question: did it ever happen that a program that has been rated number 1 on
the SSDF had been found later to be clearly weaker than its peers?
I don't remember it happened.
But I remember that it happened often that an inferior program won a
tournament...
>-operator mistakes have been found, and most likely there are more mistakes.
The tension in the tournaments is tiring and operator mistakes are not uncommon.
Saying that these errors are fixed in the right way is just your guess.
I don't even talk about hardware discrepencies and hardware problems:
1) the computers used are not all the same, so you don't compare programs in a
tournament, you compare the harware/software combination.
2) in some tournaments, hardware problems disturb the results even more. I
remember in Paris, 1997, a sponsor had promissed 233MHz computers (or was it
266MHz). But he was only able to provide 200MHz (or was it 233MHz) computers. A
number of competitors have been hurt by this problem.
So long for the fairness and accuracy of results in tournaments.
>-It is too easy for a tester to cheat (for example, playing 22 games and then
>removing two losses for its favorite program, reporting a 20-game match). I
>believe almost all the testers are honest.
You could as well say that as you don't attend the tournaments yourself you
don't trust the results. Maybe they are not sincere...
We have indeed no reason to believe the SSDF guys. But year after year they have
published results that have been proven to be reliable. 15 years of reasonnably
accurate results is enough for me to give their results the credit they deserve.
Of course they can improve, but they are anyway our most reliable source
concerning chess computer playing strength. Since a long time.
>-SSDF rating list gives very wide margins of error, which actually prevent
>anybody from knowing certainly which is "best".
So what? Tournament results have even bigger margin of errors, sometimes the
error is so big that it is not possible to say which program is the best,
because the one that has been ranked last could, given the error margin, end up
in the first place.
Once again an argument in favor of the SSDF.
>-As I have said before, the selection of opponents is quite arbitrary. Harald
>Faber has said that he can make any top program number one by correctly choosing
>the opponents, and I believe him.
It is even worse in tournaments. Have you followed the last WMCCC? There were
very weak programs allowed to compete, and this has caused some unfairness and
polemics.
It WOULD be possible to bias the result of the SSDF by a big cheat on the choice
of the opponents, but it would be very obvious, everybody would notice it. They
are not doing that. The list of opponents of a given program is always
published.
Can you find cheats in the choice of opponents and point them out?
>>The author will not be with you in your home and help YOUR copy of the program
>>to play better.
>
>Very true! I might be influenced by the simple fact that I do not have a
>computer, and obviously I also do not have any chess program either.
>
>>Generally, you know, the author is not provided in the program's
>>package.
>
>And I personally would not like to get one (unless Katja Riemann or Claudia
>Schiffer write a chess engine, even if it plays weak chess).
Here we have a strong point of agreement. Actually I don't know who Katja is,
but I trust you blindly! :)
>>If you want the best program you can run on YOUR computer, the SSDF tells you
>>much better than tournament games.
>
>I have pointed several problems with SSDF testing, but I do not want to insist
>on them (I am not against SSDF or against private testing at all).
>I only hope that by now it is clear where I am standing, what the differences
>between your position and mine are, and that the readers from this thread are at
>least amused.
I think it would be better for you to simply say that you religiously believe in
tournaments. All the arguments you can provide to belittle the value of the SSDF
actually can be used to point out the weaknesses of tournaments results...
Christophe
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.