Author: Matthew Hull
Date: 13:17:19 09/12/03
Go up one level in this thread
On September 12, 2003 at 15:56:22, Bob Durrett wrote: >On September 12, 2003 at 13:39:55, Uri Blass wrote: > >>On September 12, 2003 at 12:56:56, Bob Durrett wrote: >> >>>On September 12, 2003 at 11:00:58, Matthew Hull wrote: >>> >>>>On September 12, 2003 at 08:43:02, Bob Durrett wrote: >>>> >>>>>On September 11, 2003 at 21:43:05, Robert Hyatt wrote: >>>>> >>>>>>On September 11, 2003 at 21:19:30, Bob Durrett wrote: >>>>>> >>>>>>>I have been comparing the top engines [on the same computer] by evaluating the >>>>>>>same position, sequentially, with each engine. Many examples so far, but in >>>>>>>every case the time required to get to a given search depth is MUCH longer for >>>>>>>Crafty. Why is that? Also, does it matter in terms of playing strength? >>>>>> >>>>>>Why is that? Different pruning rules. Commercial programs are doing some >>>>>>interesting forward pruning stuff. The only forward pruning I am using is >>>>>>null-move. >>>>> >>>>>I guess the "theory" is that successful pruning [forward or otherwise] is that >>>>>time is not wasted in evaluating inconsequential [i.e. irrelevant in the search >>>>>for the "best move"] moves and branches. >>>>> >>>>>The "64 Million Dollar Question" is: "How does a programmer know that his >>>>>pruning strategy is good?" The technical goal of such strategies must be to >>>>>decrease the time to reach the correct move while holding the probability of >>>>>error to within some acceptable limits. >>>>> >>>>>The answer must be in extensive testing. Maybe, too, some theoretical >>>>>considerations might help with finding the answer. Perhaps, ultimately, the >>>>>answer must be expressed statistically? >>>> >>>> >>>>> >>>>>Clearly, if a good pruning strategy exists, then the programmer is remiss if >>>>>he/she does not find and use it. >>>> >>>> >>>> >>>>Hey now, Bob D. Are you scolding the professor? I've always found it prudent >>>>not to underestimate (much less lecture) those with experience. >>>> >>>>MH >>>> >>>>P.S. "Old age and treachery will defeat youth and skill." >>> >>>I am older than Bob Hyatt, so I have more "experience" than he does. : ) >>> >>>On a more serious note, I am a USER of chess software as I have said here many >>>times. Currently, I am embroiled in doing a comparison of the top engines >>>[Shredder 7.0.4, Tiger 15, Deep Junior 8 (used on a single processor PC), Fritz >>>8, and Crafty 19.03.] This comparison is from a USER PERSPECTIVE. In each >>>case, I am using the engine as an analysis engine running under Chessbase 8 so >>>that differences in the native GUIs are transparent. >>> >>>The method of comparison is to analyze games [of a friend] where each move of >>>the friend's game is evaluated by each engine. I am allowing each engine to >>>reach a depth of no less than 16 for each move. I have analyzed about ten games >>>this way. [That's a lot of moves!] >>> >>>Certain patterns are becoming apparent: >>> >>>(1) It takes Crafty a lot longer to reach depth = 16 than the others. >>> >>>(2) Shredder consistently gives position evaluations which are INCONSISTENT >>>with those given by the other engines. [Shredder gives larger absolute values.] >> >>It means nothing. >>The meaning of +2 of shredder is simply different than the meaning of +2 of >>other engines. > >Uri, that IS useful information for me if you mean that the choice of the move >is not impacted. It confirms my intuition in the matter. > >There is another consideration. As a user, I wish to know whether a position >should be evaluated as -+, -/+, -/=, =, +/=, +/-, or +-. These symbols are in >widespread use in the published literature and their definitions have been >around long before chess engines and have nothing to do with the way chess >engines work. > >The programmer of each engine [or GUI?] must decide how to convert the numerical >position evaluation score to the appropriate evaluation symbol. Perhaps there >has been an unwritten convention adopted by most engine/GUI designers? For >whatever reason, Shredder 7.0.4 often says a position is +- when the other >engines are saying +/-. Similarly, Shredder says +/- when the other engines are >saying +/=, =, or even -/=. This is a real problem for a USER of chess analysis >engines because the user's subsequent decisions regarding the use of the >moves/lines can be thrown off the mark. I think you might get a similar variation from different Grandmasters, too. Yes? Just look at the controversy over DB/Kasparov games. You have GMs and IMs all over the map on some of those games. MH > >I use the analysis engine as a starting point and then try to understand the >"inner workings" of the position by additional evaluations and trying to explain >to myself [and others] what is going on in the position. > >A chess engine which leads me astray right from the very beginning is worse than >useless! > >Bob D. > > >> >>It can be easily fixed by dividing the evaluation of shredder by a constant. >>> >>>(3) The "best move" selected by DJ is often different from that chosen by the >>>other engines. It often happens that the other engines all agree on the same >>>move. DJ is a "dark horse" in this respect. It should be noted that DJ jumps >>>to a higher depth very quickly, giving doubt as to what DJ's depth display >>>actually means [from the user perspective]. >>> >>>Bob D. >> >>Junior depth does not mean plies. >> >>Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.