Author: Bob Durrett
Date: 12:56:22 09/12/03
Go up one level in this thread
On September 12, 2003 at 13:39:55, Uri Blass wrote: >On September 12, 2003 at 12:56:56, Bob Durrett wrote: > >>On September 12, 2003 at 11:00:58, Matthew Hull wrote: >> >>>On September 12, 2003 at 08:43:02, Bob Durrett wrote: >>> >>>>On September 11, 2003 at 21:43:05, Robert Hyatt wrote: >>>> >>>>>On September 11, 2003 at 21:19:30, Bob Durrett wrote: >>>>> >>>>>>I have been comparing the top engines [on the same computer] by evaluating the >>>>>>same position, sequentially, with each engine. Many examples so far, but in >>>>>>every case the time required to get to a given search depth is MUCH longer for >>>>>>Crafty. Why is that? Also, does it matter in terms of playing strength? >>>>> >>>>>Why is that? Different pruning rules. Commercial programs are doing some >>>>>interesting forward pruning stuff. The only forward pruning I am using is >>>>>null-move. >>>> >>>>I guess the "theory" is that successful pruning [forward or otherwise] is that >>>>time is not wasted in evaluating inconsequential [i.e. irrelevant in the search >>>>for the "best move"] moves and branches. >>>> >>>>The "64 Million Dollar Question" is: "How does a programmer know that his >>>>pruning strategy is good?" The technical goal of such strategies must be to >>>>decrease the time to reach the correct move while holding the probability of >>>>error to within some acceptable limits. >>>> >>>>The answer must be in extensive testing. Maybe, too, some theoretical >>>>considerations might help with finding the answer. Perhaps, ultimately, the >>>>answer must be expressed statistically? >>> >>> >>>> >>>>Clearly, if a good pruning strategy exists, then the programmer is remiss if >>>>he/she does not find and use it. >>> >>> >>> >>>Hey now, Bob D. Are you scolding the professor? I've always found it prudent >>>not to underestimate (much less lecture) those with experience. >>> >>>MH >>> >>>P.S. "Old age and treachery will defeat youth and skill." >> >>I am older than Bob Hyatt, so I have more "experience" than he does. : ) >> >>On a more serious note, I am a USER of chess software as I have said here many >>times. Currently, I am embroiled in doing a comparison of the top engines >>[Shredder 7.0.4, Tiger 15, Deep Junior 8 (used on a single processor PC), Fritz >>8, and Crafty 19.03.] This comparison is from a USER PERSPECTIVE. In each >>case, I am using the engine as an analysis engine running under Chessbase 8 so >>that differences in the native GUIs are transparent. >> >>The method of comparison is to analyze games [of a friend] where each move of >>the friend's game is evaluated by each engine. I am allowing each engine to >>reach a depth of no less than 16 for each move. I have analyzed about ten games >>this way. [That's a lot of moves!] >> >>Certain patterns are becoming apparent: >> >>(1) It takes Crafty a lot longer to reach depth = 16 than the others. >> >>(2) Shredder consistently gives position evaluations which are INCONSISTENT >>with those given by the other engines. [Shredder gives larger absolute values.] > >It means nothing. >The meaning of +2 of shredder is simply different than the meaning of +2 of >other engines. Uri, that IS useful information for me if you mean that the choice of the move is not impacted. It confirms my intuition in the matter. There is another consideration. As a user, I wish to know whether a position should be evaluated as -+, -/+, -/=, =, +/=, +/-, or +-. These symbols are in widespread use in the published literature and their definitions have been around long before chess engines and have nothing to do with the way chess engines work. The programmer of each engine [or GUI?] must decide how to convert the numerical position evaluation score to the appropriate evaluation symbol. Perhaps there has been an unwritten convention adopted by most engine/GUI designers? For whatever reason, Shredder 7.0.4 often says a position is +- when the other engines are saying +/-. Similarly, Shredder says +/- when the other engines are saying +/=, =, or even -/=. This is a real problem for a USER of chess analysis engines because the user's subsequent decisions regarding the use of the moves/lines can be thrown off the mark. I use the analysis engine as a starting point and then try to understand the "inner workings" of the position by additional evaluations and trying to explain to myself [and others] what is going on in the position. A chess engine which leads me astray right from the very beginning is worse than useless! Bob D. > >It can be easily fixed by dividing the evaluation of shredder by a constant. >> >>(3) The "best move" selected by DJ is often different from that chosen by the >>other engines. It often happens that the other engines all agree on the same >>move. DJ is a "dark horse" in this respect. It should be noted that DJ jumps >>to a higher depth very quickly, giving doubt as to what DJ's depth display >>actually means [from the user perspective]. >> >>Bob D. > >Junior depth does not mean plies. > >Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.