Author: Christophe Theron
Date: 17:06:37 11/18/99
Go up one level in this thread
On November 18, 1999 at 18:43:26, Micheal Cummings wrote:
>On November 18, 1999 at 15:15:12, Christophe Theron wrote:
>
>>On November 18, 1999 at 03:09:38, Micheal Cummings wrote:
>>
>>>On November 18, 1999 at 02:00:32, Christophe Theron wrote:
>>>
>>>>On November 17, 1999 at 19:11:26, Micheal Cummings wrote:
>>>>
>>>>>On November 17, 1999 at 17:26:00, Christophe Theron wrote:
>>>>>
>>>>>>On November 17, 1999 at 01:36:32, Micheal Cummings wrote:
>>>>>>
>>>>>>>On November 17, 1999 at 01:18:49, Jouni Uski wrote:
>>>>>>>
>>>>>>>>Yes, if Chess Tiger gets 2726 rating from SSDF games, it goes to
>>>>>>>>place 4. in PCA world list (after Kramnik and BEFORE Anand). Ridiculous or ??
>>>>>>>>
>>>>>>>>Jouni
>>>>>>>
>>>>>>>I will be paying little attention to the SSDF, until they can play more programs
>>>>>>>than the 4 they have only played on the faster hardware, I will take the results
>>>>>>>with a grain of salt. To me I will deduct the rankings from the 200MMX machine
>>>>>>>results.
>>>>>>
>>>>>>Do you mean you want to calculate the rating of the programs they have tested on
>>>>>>K6 by taking only the results of these programs against P200MMX programs?
>>>>>>Because you think programs on 200MMX have more reliable ratings?
>>>>>>
>>>>>>If you do, can you post the list please?
>>>>>>
>>>>>>
>>>>>> Christophe
>>>>>
>>>>>Howdy, No I want them to play more than 4 programs on the 450 machines, I think
>>>>>it would be clearer to play the top 10 programs on the 450 machines and rank
>>>>>them on that, rather than play a select group of programs and ranking them using
>>>>>calulations between faster and slower hardward.
>>>>
>>>>
>>>>They need physical time to do it. I'm sure that's what they want to do anyway.
>>>>
>>>>
>>>>
>>>>>Most of the recent programs have all been played on the 200MMX machines so we
>>>>>pretty much know how good they are in relation to each other.
>>>>>
>>>>>For example CM6K is better on the 200MMX than all the other programs, And I
>>>>>think is way better than Nimzo which is No 4 on the 450 machines on the last
>>>>>list ahead of CM6K. This is not right in my opinion, where would CM6K be if it
>>>>>was played on faster hardware as the other 4 programs.
>>>>
>>>>
>>>>I would have been surprised if you had not talked about CM6000!!! :)
>>>>
>>>>There is no doubt that CM6000 is very strong, I'll not deny it.
>>>>
>>>>But if it is not listed with the fastest hardware, it's Minscape's fault.
>>>>
>>>>If they were interested to prove the strength of their program, they would have
>>>>provided some kind of help to play automatic games.
>>>>
>>>>
>>>>
>>>>>That is what I mean, at the moment the list is unfair.
>>>>
>>>>So what should they do? Publish nothing until all 10 programs have played more
>>>>than 300 games each?
>>>>
>>>>Next list: mid 2000.
>>>>
>>>>
>>>>
>>>>>And with this ranking thing for working out the elo of programs, if all was fair
>>>>>why are the top four programs that far in front, either way I cannot explain
>>>>>what I mean in the rankings, it just seems like I have always said that it is
>>>>>selectively unfair
>>>>
>>>>
>>>>You did not say that when they have made the effort to play 184 games BY HAND
>>>>with CM6000. Do you realize how long it took? Would you like to do that
>>>>yourself? If the games take only 5 hours each, that makes 920 hours of manual
>>>>operation, or more than 38 days of your life, 24 hours a day, operating CM6000.
>>>>If you do it only 8 hours a day, be prepared to operate CM6000 for 115 days
>>>>(almost 4 months). Are you ready to give 4 months of your salary just to know
>>>>how strong CM6000 is? In a sense, they did it.
>>>>
>>>>They have made this tremendous effort just for your favorite chess program, this
>>>>allowed it to be on top of the list, and now you say they are unfair?
>>>
>>>Well actually YES :) So there are great logistics involved in getting the
>>>results. Just because something takes a long time and hard to do, does not make
>>>the results fair, do you agree ??
>>>
>>>Plus you are giving CM6K as only one example of my larger argument, there are
>>>plenty of other programs with autoplayers they can use. The results are unfair
>>>for Shredder and any other program that was not one of those four tested on the
>>>previous list.
>>>
>>>Even though someone tries and does their best does not make things fair. If you
>>>live in a flat and needed some plumbing done, and the landlord tried to fix it
>>>himself the best he could and that night the pipes burst again and wrecked your
>>>computer system and he said he tried his best. Is it fair that he did a
>>>substandard job when he should of got a real plumber, even though he might he
>>>your friend and a great guy ??
>>>
>>>Just doing the best you can is not enough sometimes. Yes you can appriciate all
>>>they can do and what they have done. But if it leads to results that are
>>>lacking, then you have to start thinking in a different line.
>>>
>>>The guys from mindscape got paid to create CM7K, I am sure they did their best
>>>and tried their hardest, but the program IMHO is a piece of crap.
>>>
>>>The reson why I pick on the SSDF, is that I have seen companies and sites use
>>>SSDF rankings to help sell the product. And when this takes place the SSDF has
>>>to make sure that the info they release is clear and close enough to being
>>>correct.
>>>
>>>And yes I said when the list was released, that they should not have included
>>>those four programs in that list till they had tested more. (CM6K or no CM6K).
>>>
>>>Lets say Tiger tops the SSDF on a 450 machine ahead of the other 4 already
>>>tested. What about CM6K, what about Shredder ??, yes there are logistics
>>>involved, but does it make it fair ??
>>
>>
>>Each program represents a company or a programmer. They don't come out of the
>>blue. It's the job of the companies, or of the programmers, to help in order to
>>have their programs rated if they feel they have a chance to be in good place on
>>the list.
>>
>>Imagine I produce a program, pretend that it is playing at a 2800 level, but
>>make no effort to make it easy to use. No autoplayer, moves have to be entered
>>by the keyboard, in cryptic format (wj/i7 means e2-e4 for example), and the
>>board is not displayed on the screen.
>>
>>Would you call the SSDF unfair if they don't test this piece of crap?
>>
>>Minscape did not provide autoplaying capabilities for ChessMaster6000. The SSDF
>>guys have been more than nice to play this program manually.
>>
>>I'm not sure people are interested to see an old version of Shredder tested. I
>>suppose most people would like to see Shredder 4 tested in the next list.
>>
>>Unfortunately Millenium does not want Shredder 4 to be listed.
>>
>>So it's a little bit easy to jump on the SSDF.
>>
>>
>>
>> Christophe
>
>
>All your points I take. But take it simpler still. SSDF produces a list ranking
>chess programs.
>
>1. Not all programs are tested
>2. Calculations to give results for different level hardware
>
>
>In a perfect list world, all programs should be tested, and all programs tested
>on equal hardware to give equal chance.
>
>That would be fair. It really does not matter the other logistics involved. Fair
>testing is very easy, how easy it is to obtain is irrelvant.
>
>So my point is that the SSDF testing is not fair, people accept it for many
>reasons, but I will argue with you till the cows come home that you can find
>anyone who can tell me that it is fair.
>
>As you said yourself, you want to see a newer program tested, well having the
>older one is not fair, you have just accepted it.
>
>I think we will have to agree to disagree here, I am sure you know its not fair,
>but you are looking at it from an angle which you see them doing their best and
>accept the data they have.
OK. I understand.
So maybe we can find a way to agree somehow.
Let's say the list is not perfect due to lack of physical resources, that it is
necessarily unfair because every program cannot be tested on every hardware, but
that the people doing the list are trying to make it as fair as possible given
the physical limitations they have to deal with.
If we at least agree that they are trying to do their best and that it's not
easy for them to please everyone, well that's a good compromise.
Christophe
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.