Author: Kurt Utzinger
Date: 11:19:12 07/17/05
Go up one level in this thread
On July 17, 2005 at 05:22:47, Sandro Necchi wrote:
>I have been laughing a lot (maybe crying on the ignorance would have been more
>appropriate?)reading many wrong statements about testing and Elo lists.
>
>so, for those who are new and do not know, SSDF list is the best for the
>following reasons:
>
>1. They use 2 computers and the program complete with own book and ETG, with own
>gui and best setting as suggested by the programmer.
>2. They use long time controls (40/2h 20/1h; international level) only.
>3. They use the same hardware for all programs.
>4. They use a very wide range of programs and not only the new ones to get more
>reliable results.
>5. Ponder on and learning are activated.
Nothing against the SSDF list and the people who are
testing. They are doing a great job. Nevertheless, I
would never go so far saying it's the only reliable
and therefore the best list.
Kurt
>Now, even if some people do not agree, the use of own book is the best because
>that book has been developed specifically for that engine and in some cases the
>engine has been developed specifically on that book too. This means that the use
>of a different book and the same for all programs would damage or favor some
>programs over others. Even "neutral" books would do the same as they may include
>variations which are not "compatible" with some sophisticated programs and be OK
>for others.
>I know some people do not agre on this, but this is their problem...
Here I can't agree. I am no book expert but think that the
the orginal books have by way of exception been developed
specifically for that engine. Most of these books contain too
much lines with either holes in them or just including lines
simply based on performance of (top) GM's without considering
if the engine does understand the position. A strong program
will not be harmed by a "neutral book" and Shredder is a very
good example winning all matches using 5moves.ctg, remis.ctg
or pre-defined openings like Nunn2, Noomen select and so on.
Kurt
>The use of a "std" gui again would favor some and damage others as well, so it
>is not advisable...
Is there any data about this?
Kurt
>
>The use of long time controls is the best to really check the max potentiality
>of a program. It is true that the hardware used by SSDF is not updated, but 2 or
>3 times faster hardware would not change much even if some programs may benefit
>a little more than others (a small Elo difference).
I agree although we should accept that testing with
differenct time controls is the only way to get an
overall expression about strength and weakness of
a chess program.
Kurt
>
>Some people claim better programs against humans then computers. These are pure
>lies as if you play better you play better against anybody. These are more
>"commercial" statements than true ones...of course there is no relationship on
>Elo figures on the SSDF list with those against humans, but a stronger program
>here would do better against humans too. The problem is that in order to achive
>reliable results there is a need of very many games. A few game may be
>confusing.
Whilst it may indeed be very difficult to say that program X
is doing better vs humans than program Y - due to insufficient
data, say too less games - I nevertheless think that there are
differences. I have tried in many, many games to get draws vs
the best engines with my well known and boring style and I
I had always (and still have) the feeling that it is much easier
to achieve this vs Fritz and ChessTiger than against Hiarcs or
Gandalf. But as already said: too less data to "prove" anything.
Kurt
>
>Thanks to the use of 2 computers one can also test against old program too. This
>may seems useless, but it is not.
>
>Since the goal of SSDF list is to tell how strong is a new program to use the
>best settings and learning is a must too because the user can use the same and
>would like to know how strong is that program with best settings etc...
>If some programs do not have learning features and/or good ones it is their
>problem so they have to be penalized on that. The use of these options would do
>this.
>
>So, anybody can test in a different way as they wish, but to claim that system
>is better or replacing the SSDF system is pure nonsense!
There are not many who claim that their system would be
better than SSDF. On the other hand we should also accept
others testing methods. Personally I do not like the use
of (original) big books and learning. I am more interested
in the naked engine strength under neutral conditions and
as analysing tool. And for me one thing is clear: if you always
use the same neutral books/positions for testing you will
get much easier/faster comparable results between old/new
version of a program. And finally: it's anyway more than
boring to see an engine start thinking at move 29.
Kurt
>
>Sandro
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.