Author: José Carlos
Date: 00:02:28 10/19/01
Go up one level in this thread
On October 18, 2001 at 20:29:43, Christophe Theron wrote: >On October 18, 2001 at 16:11:01, José Carlos wrote: > >>On October 18, 2001 at 14:49:01, Christophe Theron wrote: >> >>>On October 18, 2001 at 07:22:26, José Carlos wrote: >>> >>>>On October 17, 2001 at 15:28:30, Christophe Theron wrote: >>>> >>>>>On October 17, 2001 at 14:56:16, Chris Taylor wrote: >>>>> >>>>>>I have played Fritz 3.10 against newish progams. >>>>>>F3 ran on an AMD 1200 inside F6 gui, using General.ctg >>>>>>1 x 1200 played against the PIII 733, the other played against the AMD 800 >>>>>>The opponants >>>>>> >>>>>>PIII 733 Gambit Tiger 2.0, WcraftyP3 1811. This was offered as optimised for >>>>>>P3... >>>>>>Tiger ran with its own book, in F6 gui. Wcrafty's book is from Bob's site. >>>>>>Crafty ran under Remi Coulom's wbenging0047, with full auto232. >>>>>> >>>>>>AMD 800 Junior 4.6, Junior 6. Both versions of Junior ran under F6 gui, with >>>>>>Junior.ctg >>>>>> >>>>>> >>>>>>One of the oldest programs, that can run on my newest machines. Versus some >>>>>>newish stuff. >>>>>> >>>>>>Game in 1 hour... >>>>>> >>>>>>Junior 4.6 v Fritz 3.10 4-2 >>>>>>Junior +3 -1 =2 >>>>>> >>>>>>Junior 6 v Fritz 3.10 4½-1½ >>>>>>Junior +3 -0 =3 >>>>>> >>>>>>Gambit Tiger 2.0 v Fritz 3.10 5½-½ >>>>>>Tiger +5½ -0 =1 >>>>>> >>>>>>WcraftyP3 1811 v Fritz 3.10 4-2 >>>>>>Crafty +3 -1 =2 >>>>>> >>>>>>Just a small sample of games. Anyone wanting the pgn file of 24 games is >>>>>>welcome to it. >>>>>> >>>>>>Chris Taylor >>>>> >>>>> >>>>> >>>>>Is anybody still wondering if there have been progress in chess programming in >>>>>the last years? >>>>> >>>>> >>>>> >>>>> Christophe >>>> >>>> I don't say there isn't. It'd be absurd. But a couple of remarks: >>>> >>>> a. Your statement seems to imply that such a small number of games proves "the >>>>progress". I guess I got you wrong because you always claim a lot of games are >>>>needed to make any conclusion. >>> >>> >>> >>>The overall result of the match is 18-6, that is a 75% win for the newer >>>software on inferior hardware. >>> >>>I look at my statistical tables and I see that this result is significant with >>>90% confidence. The error margin for 24 games is under 15% with 90% confidence, >>>and here we got a result of 25% above 50%. >>> >>>So I'm 90% sure that the new software on slower hardware is better than Fritz3 >>>on the faster hardware. >>> >>>I could even say that new software on slower hardware is *significantly* better >>>than Fritz3 on faster hardware, not even talking about what would happen on >>>equal hardware. >>> >>> >>>It's not as simple as "a lot of games are needed to make any conclusion". >>> >>>It all depends on the number of games and the difference between the opponents. >>>When the difference in elo is big (as it is the case here), you do not need a >>>huge number of games. >>> >>>However when the difference is small, you need LOTS of games. >>> >>>I have already published several times the statistical tables that I am using, >>>but apparently nobody has any interest for this... >> >> You are right. I don't remember your tables. Nor I need them now. Flip a coin >>24 times: anything can happen. > > > >Anything can happen, yes. > >But if we start to bet and I have some knowledge about statistics and you don't, >then I'm going to win all your money. So maybe you should care a little about >the tables below: > > >----------------------------------------------------------------- >Reliability of chess matches >(assuming each opponent has 1/3 chances to win, 1/3 to loose and 1/3 to draw) > >90% confidence >Games %err+/- elo+/- > 10 20 140pts > 20 15 105pts > 25 14 98pts > 30 12 63pts > 40 10 70pts > 50 9 56pts > 100 6.5 35pts > 200 4.72 33pts > 400 3.34 23pts > 600 2.66 19pts > 800 2.39 17pts > 1000 2.12 15pts > 1200 2.00 14pts > 1400 1.81 13pts > 1600 1.66 12pts > >80% confidence >Games %err+/- elo+/- > 10 15 105pts > 20 11 77pts > 25 10 70pts > 30 9 63pts > 40 8 56pts > 50 7 49pts > 100 5.0 35pts > 200 3.75 26pts > 400 2.60 18pts > 600 2.15 15pts > 800 1.86 13pts > 1000 1.66 12pts > 1200 1.46 10pts > 1400 1.40 10pts > 1600 1.34 9pts > >70% confidence >Games %err+/- elo+/- > 10 15 105pts > 20 10 70pts > 25 8 56pts > 30 8 56pts > 40 6.3 44pts > 50 6.0 42pts > 100 4.0 28pts > 200 3.0 21pts > 400 2.2 15pts > 600 1.7 12pts > 800 1.5 11pts > 1000 1.3 9pts > 1200 1.24 9pts > 1400 1.14 8pts > 1600 1.04 7pts > >----------------------------------------------------------------- > >These tables are slightly wrong because of the 1/3-1/3-1/3 assumption mentionned >above, but they are accurate enough for most uses. > > > > > > >>>> b. If by "progress in chess programming" you mean "software-only progress" >>> >>> >>>Yes I mean "software only". >> >> I'd be really interested in understanding this concept. Is is about theorical >>algorithms? > > >Yes, mainly. > > > >>Otherwise it's a mistery to me. Software is absolutely >>hardware-dependent, and I'm not talking of Mhz at all. I'm speaking of things >>such as: >> -If I have small memory I need to write a small program for it runs fast so I >>miss the great possibilities free memory gives me. >> -If I have a great graphical card, I'll write my application (I'm not talking >>about chess, but in general) with fancy images and graphical effects. >> -If I have a slow server for my database, I'll try to write very simple >>queries for not overloading the server. >> -If I have a very fast LAN, I can move a lot of data to the clients and make >>them run faster. Otherwise, I try to minimize netware traffic. >> -If I have a fast and big L2 cache I'll write my program to fit in it and it >>will run fine there, but very slow in a machine with small cache. >> >> I can give thousand if examples. Every single line of code I write, I do it >>knowing what system/s its gonna run in. I know my programs can't run in ZX >>Spectrums, and I don't care. >> >> So please, provide examples of pieces of code that are _absolutely_ >>independent from hardware. > > >Nothing is absolutely independant of hardware, you are right. > >But here we were talking about PC platforms ranging from 486 to fast Athlon or >P4. That narrows the scope of what "hardware" is. > >And we were talking about chess engines, not about database programs. > >On this range of platforms, I can give you one example of a piece of code which >does not care much about the hardware: the Chess Tiger engine. > >It runs fine on the aforementionned PC platform, from the 386sx 20MHz that I use >often, to the fastest PC hardware you can find. > >When I say "it runs fine", I mean that I do not know of any optimization that >would significantly improve the performances of the engine on a given machine >inside the platform range, and would not work on the rest of the platform. > >I can only think about optimizations that would give a very minor improvement in >performances (say 5%). > >I do not say that one cannot write a chess engine that would perform much better >on one machine on the platform and not on the others, but my opinion about this >code is that it would suck, because it is possible to write a chess engine that >performs as well, or only marginally worse, on ALL the machines of the given >platform. > > > > > >>>>(whatever that means -that concept is beyond my understanding, because I always >>>>optimize my code for a certain kind of hardware-), no conclusion can be made >>>>without testing in both new and old hardware. For example, if Fritz 3 is to be >>>>evaluated, 486-33 would be a good hardware to test the programs in. Then, >>>>comparing results in both kind of hardware would yield more interesting >>>>conclusions. >>> >>> >>>You overestimate what can be achieved by optimizing for a given hardware. >>> >>>We are talking about a difference of 5% in speed here, depending on what >>>processor you have optimized for, and this speed difference corresponds to a 5 >>>elo points difference. This is negligible because you would need a huge number >>>of games to detect such a difference. >> >> Answered above. I'm not talking of that "optimization" only. That's a small >>part of the thing. >> Anyway, don't get me wrong: I agree that software has improved. But I think >>_most_ of these improvements are allowed by hardware improvements. > > > >It's definitely wrong in the case of my engine. > >But I don't want to be too much negative about this. You are right in sense: >having at my disposal a much more powerful hardware has allowed me to develop my >program faster (because the compilation times are very short now), run automatic >tests in minutes (when they would have needed years to complete with older >hardware), and overall help me to work faster. > >But all this time saved because of these better tools results in a chess engine >that is improved on ALL the given platform (from 486 to Athlon), not just the >class of machines it has been developped on recently (I'm currently using a K6-2 >450MHz to develop). > > > > Christophe Ok. It seems we agree in the background, but disagree in the surface. Software is much better now, true. Hardware has helped software developement, true. The point we disagree in can (I think) be said in a line: I believe that, if Frans would have back then the kind of hardware we have now, Fritz 3 would be much stronger, much closer to Fritz 7. And for the same reason, if we had now exactly the same machines as at that time, we couldn't have done many of the things we do now. That's all. José C.
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.