Author: Matthew Hull
Date: 11:05:41 09/12/03
Go up one level in this thread
On September 12, 2003 at 12:56:56, Bob Durrett wrote: >On September 12, 2003 at 11:00:58, Matthew Hull wrote: > >>On September 12, 2003 at 08:43:02, Bob Durrett wrote: >> >>>On September 11, 2003 at 21:43:05, Robert Hyatt wrote: >>> >>>>On September 11, 2003 at 21:19:30, Bob Durrett wrote: >>>> >>>>>I have been comparing the top engines [on the same computer] by evaluating the >>>>>same position, sequentially, with each engine. Many examples so far, but in >>>>>every case the time required to get to a given search depth is MUCH longer for >>>>>Crafty. Why is that? Also, does it matter in terms of playing strength? >>>> >>>>Why is that? Different pruning rules. Commercial programs are doing some >>>>interesting forward pruning stuff. The only forward pruning I am using is >>>>null-move. >>> >>>I guess the "theory" is that successful pruning [forward or otherwise] is that >>>time is not wasted in evaluating inconsequential [i.e. irrelevant in the search >>>for the "best move"] moves and branches. >>> >>>The "64 Million Dollar Question" is: "How does a programmer know that his >>>pruning strategy is good?" The technical goal of such strategies must be to >>>decrease the time to reach the correct move while holding the probability of >>>error to within some acceptable limits. >>> >>>The answer must be in extensive testing. Maybe, too, some theoretical >>>considerations might help with finding the answer. Perhaps, ultimately, the >>>answer must be expressed statistically? >> >> >>> >>>Clearly, if a good pruning strategy exists, then the programmer is remiss if >>>he/she does not find and use it. >> >> >> >>Hey now, Bob D. Are you scolding the professor? I've always found it prudent >>not to underestimate (much less lecture) those with experience. >> >>MH >> >>P.S. "Old age and treachery will defeat youth and skill." > >I am older than Bob Hyatt, so I have more "experience" than he does. : ) > >On a more serious note, I am a USER of chess software as I have said here many >times. Currently, I am embroiled in doing a comparison of the top engines >[Shredder 7.0.4, Tiger 15, Deep Junior 8 (used on a single processor PC), Fritz >8, and Crafty 19.03.] This comparison is from a USER PERSPECTIVE. In each >case, I am using the engine as an analysis engine running under Chessbase 8 so >that differences in the native GUIs are transparent. > >The method of comparison is to analyze games [of a friend] where each move of >the friend's game is evaluated by each engine. I am allowing each engine to >reach a depth of no less than 16 for each move. I have analyzed about ten games >this way. [That's a lot of moves!] > >Certain patterns are becoming apparent: > >(1) It takes Crafty a lot longer to reach depth = 16 than the others. Two issues (not that I'm a real expert, but from what I've seen here): #1 It is not depth, necessarily, but whether the program is on the correct path through the "thicket". You may or may not remember the bogus criticisms of Deep Blue which only searched 12 ply on average. What got missed is how much of that was full width (very expensive) but with very deep extensions. Full width means "never having to say you're sorry", or basically you eliminate pruning mistakes that can happen with deeper searchers. #2 Also, in a uniform platform comparison (what you are doing in effect), it comes down to which program gets the most smarts out of each cycle of cpu time. A program like Crafty, which is designed for multi-platform compatibility/portability, the smarts-per-cycle will suffer in comparison to x86-only optimized programs. Crafty tries to make up for this with selected x86 assembly (for PC compile targets) and with SMP capability (throw multiple CPUs at it). That's one reason why Crafty will likely never top the SSDF rating list because it's basically getting fewer CPU cycles than it's opponents due to it's portable design tradeoffs. Hope that illuminates more than it confuses. MH > >(2) Shredder consistently gives position evaluations which are INCONSISTENT >with those given by the other engines. [Shredder gives larger absolute values.] > >(3) The "best move" selected by DJ is often different from that chosen by the >other engines. It often happens that the other engines all agree on the same >move. DJ is a "dark horse" in this respect. It should be noted that DJ jumps >to a higher depth very quickly, giving doubt as to what DJ's depth display >actually means [from the user perspective]. > >Bob D. > > >> >> >> >>> >>>Bob D. >>> >>> >>> >>>> (I assume you are not comparing junior in this mix since junior >>>>has a different meaning for "ply" than the rest of us). >>>> >>>> >>>> >>>>> >>>>>Could it be that Crafty's findings at depth = 12 are just as good as Junior's >>>>>findings at depth = 16? >>>> >>>>You simply can't compare depths. IE compare chessmaster to junior and then >>>>decide which is better based on that. :) >>>> >>>> >>>> >>>> >>>>> >>>>>If Crafty could find a way to do just as much useful computation in half the >>>>>time [on the same computer], it would play better. Right? [Obviously.] >>>>> >>>>>Maybe Crafty 19.03 is wasting too much time on non-essential >>>>>computations!!!!!!!!!!!! >>>>> >>>>>Bob D.
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.