Author: Robert Hyatt
Date: 10:03:00 09/20/04
Go up one level in this thread
On September 20, 2004 at 12:55:07, Scott Gasch wrote: >Case in point: a couple of years ago someone posted scores from the pros on >ECM/10. Almost all of them were around 700/879 except one notable (to me) >exception: Shredder was down around 640. (Note: I'm working from memory here >and I don't know what hardware was involved). But Shredder is definitely a very >strong engine, probably one of the strongest engines out there. > >If you look at test suite results (which I do to a point, because it's a more >reliable metric than "I think the engine played well today") you'll notice some >percentage of "solved" positions where picked because of luck. The engine >didn't see the deep mate / tactic, it just liked the bishop on that square. >Tweak your eval a little and suddenly you "lose" 10 solutions. Did you really >lose anything? Probably not. > >The real question is how to emperically measure chess engine progress... >self-play? Automated tournament vs. other opponent? Test suite results? ICC >rating? I don't have an answer but I do self-play and test suites, personally. >And I don't put a whole lot of faith in the suites... > >Scott The most reliable metric is to play strong opponents and look at the logs carefully. Sometimes a win is bad as your program played poorly and lucked out.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.