Author: Don Dailey
Date: 18:49:57 01/25/99
Go up one level in this thread
On January 25, 1999 at 20:51:09, Robert Hyatt wrote: >On January 25, 1999 at 20:15:52, Don Dailey wrote: > >>On January 25, 1999 at 13:42:38, Bruce Moreland wrote: >> >>> >>>On January 25, 1999 at 12:19:09, Will Singleton wrote: >>> >>>>I think Bruce has posted the games, and I am sure we'll get some interesting >>>>data. I'll even run it through my program, just for fun. >>> >>>Please do. If you can't deal with the word-wrap on the EPD files that I posted, >>>please send mail to brucemo@seanet.com and I'll send you all 22 files as either >>>attachments or inlined into the mail, as you prefer. >>> >>>I think that if enough people do this it will serve two purposes: >>> >>>1) We will clear up this "all the moves are the same" allegation by Bob. >>>2) We will know once and for all how similar programs are to each other. I have >>>no idea what percentage of moves will be reproduceable, it would be interesting >>>to find out. >>> >>>bruce >> >>We need someone other than Bob to test on different versions of Crafty >>too. I completely trust Bob, but for scientific reasons this needs >>to be verified independantly. The test must match the hardware, so >>it needs to be run on a Pentium 400 for at least 10 minutes. >> >>- Don > > >I think the 'me not testing' is a good idea, particularly since I already did it >on the three games Vincent sent. But the hardware is a big issue, because in >that event Bionic played on a dual CPU overclocked to something over 500mhz. > >The SMP search alone will produce non-deterministic results with enough >frequency to cause consternation. And then there is crafty's time allocation, >which means that on some moves it could take 10+ minutes on a 'whim'. I think the interesting data comes from COMPARING other programs. If Bionic matches Crafty 94% of the time (which is a lot because this might mean only 2 or 3 mismatches per game) but no other program does better than 50%, this is pretty damning. Your observation about parallel nondeterminism matches my results too. You can not only get a mismatch at a given search depth, but you can get significant time differences. On a large number of processors you can get 2 or 3 to 1 (or even more) with repeat testing of the same positions, but the effect is of course smaller on less processors. I would suggest that as we report our results we also report the exact condition of the tests and the exact version of programs we use. People should do this with commercial programs too and post the results. I'm running all the games starting from the 11th move of the 1st 6 rounds (1st week.) I would like to get a hold of the executable that produced the log file of the bionic game that was posted by one of the Bionic programmers. This would help us clear this up a lot. Can we get this? - Don
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.