Author: Christophe Theron
Date: 17:29:43 10/18/01
Go up one level in this thread
On October 18, 2001 at 16:11:01, José Carlos wrote:
>On October 18, 2001 at 14:49:01, Christophe Theron wrote:
>
>>On October 18, 2001 at 07:22:26, José Carlos wrote:
>>
>>>On October 17, 2001 at 15:28:30, Christophe Theron wrote:
>>>
>>>>On October 17, 2001 at 14:56:16, Chris Taylor wrote:
>>>>
>>>>>I have played Fritz 3.10 against newish progams.
>>>>>F3 ran on an AMD 1200 inside F6 gui, using General.ctg
>>>>>1 x 1200 played against the PIII 733, the other played against the AMD 800
>>>>>The opponants
>>>>>
>>>>>PIII 733 Gambit Tiger 2.0, WcraftyP3 1811. This was offered as optimised for
>>>>>P3...
>>>>>Tiger ran with its own book, in F6 gui. Wcrafty's book is from Bob's site.
>>>>>Crafty ran under Remi Coulom's wbenging0047, with full auto232.
>>>>>
>>>>>AMD 800 Junior 4.6, Junior 6. Both versions of Junior ran under F6 gui, with
>>>>>Junior.ctg
>>>>>
>>>>>
>>>>>One of the oldest programs, that can run on my newest machines. Versus some
>>>>>newish stuff.
>>>>>
>>>>>Game in 1 hour...
>>>>>
>>>>>Junior 4.6 v Fritz 3.10 4-2
>>>>>Junior +3 -1 =2
>>>>>
>>>>>Junior 6 v Fritz 3.10 4½-1½
>>>>>Junior +3 -0 =3
>>>>>
>>>>>Gambit Tiger 2.0 v Fritz 3.10 5½-½
>>>>>Tiger +5½ -0 =1
>>>>>
>>>>>WcraftyP3 1811 v Fritz 3.10 4-2
>>>>>Crafty +3 -1 =2
>>>>>
>>>>>Just a small sample of games. Anyone wanting the pgn file of 24 games is
>>>>>welcome to it.
>>>>>
>>>>>Chris Taylor
>>>>
>>>>
>>>>
>>>>Is anybody still wondering if there have been progress in chess programming in
>>>>the last years?
>>>>
>>>>
>>>>
>>>> Christophe
>>>
>>> I don't say there isn't. It'd be absurd. But a couple of remarks:
>>>
>>> a. Your statement seems to imply that such a small number of games proves "the
>>>progress". I guess I got you wrong because you always claim a lot of games are
>>>needed to make any conclusion.
>>
>>
>>
>>The overall result of the match is 18-6, that is a 75% win for the newer
>>software on inferior hardware.
>>
>>I look at my statistical tables and I see that this result is significant with
>>90% confidence. The error margin for 24 games is under 15% with 90% confidence,
>>and here we got a result of 25% above 50%.
>>
>>So I'm 90% sure that the new software on slower hardware is better than Fritz3
>>on the faster hardware.
>>
>>I could even say that new software on slower hardware is *significantly* better
>>than Fritz3 on faster hardware, not even talking about what would happen on
>>equal hardware.
>>
>>
>>It's not as simple as "a lot of games are needed to make any conclusion".
>>
>>It all depends on the number of games and the difference between the opponents.
>>When the difference in elo is big (as it is the case here), you do not need a
>>huge number of games.
>>
>>However when the difference is small, you need LOTS of games.
>>
>>I have already published several times the statistical tables that I am using,
>>but apparently nobody has any interest for this...
>
> You are right. I don't remember your tables. Nor I need them now. Flip a coin
>24 times: anything can happen.
Anything can happen, yes.
But if we start to bet and I have some knowledge about statistics and you don't,
then I'm going to win all your money. So maybe you should care a little about
the tables below:
-----------------------------------------------------------------
Reliability of chess matches
(assuming each opponent has 1/3 chances to win, 1/3 to loose and 1/3 to draw)
90% confidence
Games %err+/- elo+/-
10 20 140pts
20 15 105pts
25 14 98pts
30 12 63pts
40 10 70pts
50 9 56pts
100 6.5 35pts
200 4.72 33pts
400 3.34 23pts
600 2.66 19pts
800 2.39 17pts
1000 2.12 15pts
1200 2.00 14pts
1400 1.81 13pts
1600 1.66 12pts
80% confidence
Games %err+/- elo+/-
10 15 105pts
20 11 77pts
25 10 70pts
30 9 63pts
40 8 56pts
50 7 49pts
100 5.0 35pts
200 3.75 26pts
400 2.60 18pts
600 2.15 15pts
800 1.86 13pts
1000 1.66 12pts
1200 1.46 10pts
1400 1.40 10pts
1600 1.34 9pts
70% confidence
Games %err+/- elo+/-
10 15 105pts
20 10 70pts
25 8 56pts
30 8 56pts
40 6.3 44pts
50 6.0 42pts
100 4.0 28pts
200 3.0 21pts
400 2.2 15pts
600 1.7 12pts
800 1.5 11pts
1000 1.3 9pts
1200 1.24 9pts
1400 1.14 8pts
1600 1.04 7pts
-----------------------------------------------------------------
These tables are slightly wrong because of the 1/3-1/3-1/3 assumption mentionned
above, but they are accurate enough for most uses.
>>> b. If by "progress in chess programming" you mean "software-only progress"
>>
>>
>>Yes I mean "software only".
>
> I'd be really interested in understanding this concept. Is is about theorical
>algorithms?
Yes, mainly.
>Otherwise it's a mistery to me. Software is absolutely
>hardware-dependent, and I'm not talking of Mhz at all. I'm speaking of things
>such as:
> -If I have small memory I need to write a small program for it runs fast so I
>miss the great possibilities free memory gives me.
> -If I have a great graphical card, I'll write my application (I'm not talking
>about chess, but in general) with fancy images and graphical effects.
> -If I have a slow server for my database, I'll try to write very simple
>queries for not overloading the server.
> -If I have a very fast LAN, I can move a lot of data to the clients and make
>them run faster. Otherwise, I try to minimize netware traffic.
> -If I have a fast and big L2 cache I'll write my program to fit in it and it
>will run fine there, but very slow in a machine with small cache.
>
> I can give thousand if examples. Every single line of code I write, I do it
>knowing what system/s its gonna run in. I know my programs can't run in ZX
>Spectrums, and I don't care.
>
> So please, provide examples of pieces of code that are _absolutely_
>independent from hardware.
Nothing is absolutely independant of hardware, you are right.
But here we were talking about PC platforms ranging from 486 to fast Athlon or
P4. That narrows the scope of what "hardware" is.
And we were talking about chess engines, not about database programs.
On this range of platforms, I can give you one example of a piece of code which
does not care much about the hardware: the Chess Tiger engine.
It runs fine on the aforementionned PC platform, from the 386sx 20MHz that I use
often, to the fastest PC hardware you can find.
When I say "it runs fine", I mean that I do not know of any optimization that
would significantly improve the performances of the engine on a given machine
inside the platform range, and would not work on the rest of the platform.
I can only think about optimizations that would give a very minor improvement in
performances (say 5%).
I do not say that one cannot write a chess engine that would perform much better
on one machine on the platform and not on the others, but my opinion about this
code is that it would suck, because it is possible to write a chess engine that
performs as well, or only marginally worse, on ALL the machines of the given
platform.
>>>(whatever that means -that concept is beyond my understanding, because I always
>>>optimize my code for a certain kind of hardware-), no conclusion can be made
>>>without testing in both new and old hardware. For example, if Fritz 3 is to be
>>>evaluated, 486-33 would be a good hardware to test the programs in. Then,
>>>comparing results in both kind of hardware would yield more interesting
>>>conclusions.
>>
>>
>>You overestimate what can be achieved by optimizing for a given hardware.
>>
>>We are talking about a difference of 5% in speed here, depending on what
>>processor you have optimized for, and this speed difference corresponds to a 5
>>elo points difference. This is negligible because you would need a huge number
>>of games to detect such a difference.
>
> Answered above. I'm not talking of that "optimization" only. That's a small
>part of the thing.
> Anyway, don't get me wrong: I agree that software has improved. But I think
>_most_ of these improvements are allowed by hardware improvements.
It's definitely wrong in the case of my engine.
But I don't want to be too much negative about this. You are right in sense:
having at my disposal a much more powerful hardware has allowed me to develop my
program faster (because the compilation times are very short now), run automatic
tests in minutes (when they would have needed years to complete with older
hardware), and overall help me to work faster.
But all this time saved because of these better tools results in a chess engine
that is improved on ALL the given platform (from 486 to Athlon), not just the
class of machines it has been developped on recently (I'm currently using a K6-2
450MHz to develop).
Christophe
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.