Author: Uri Blass
Date: 04:08:32 12/13/02
Go up one level in this thread
On December 13, 2002 at 03:17:43, Uri Blass wrote: >On December 13, 2002 at 01:08:26, Christopher A. Morgan wrote: > >> >>50 Test Positions, 15 Engines - Results, Comparisons >> >>Recently, Gian-Carlo Pascutto, I think it was, posted about 150 position >>problems, together with solutions, each with an identifying number of >>ECM.xxx(x). I took the first 51, discarded one, and ran time to solve tests on >>15 different chess engines for each of the 50 positions in the Fritz 7 GUI Tools >>–> Analysis –> Process Test Set window. Below are the results of the tests. >>The problems in FEN, as previously posted, follow the results. >> >>Some details: For each of the problems I confirmed the solution by letting a >>couple of engines run individually in infinite analysis mode for 5-10 minutes. >>The one problem discarded had three different solutions by four different >>engines. For four of the problems, numbers 14, 28, 29 and 38 I listed the text >>move as the solution together with a variation. In one case Nimzo 8 was only >>engine of four I tested with that came up with the text solution. The other >>three engines agreed on a different solution which became the variation. For >>the other three positions, multiple engines agreed on a different solution which >>became the variation. >> >>My goal was to have the majority of the engines solve every problem, so it would >>be a test of how quickly a particular engine solved a problem compared to all >>other engines in average speed of finding solutions to all problems, rather than >>running for ten minutes (maximum allowed time per position) and not finding a >>solution. >> >>Hardware: Athlon 750, 384MB RAM, 144MB RAM hashtables, except for Chess Tiger >>14, and Gambit Tiger 2 which, apparently, only allow a maximum of a 96MB >>hashtable. The times given should only be looked at in relative terms, that is >>relative to the other engines. Faster processors will get much faster times, >>but I would expect that the relative percentage differences in average speed >>should remain constant among different processors running the same problems with >>the given solutions. >> >>Problems 1, 24, 31, 33, and 41 took the most time for most engines, and a few >>were not able to find a solution for some of these in the maximum ten minutes >>allowed. > >I am surprised because 1 is one of the easiest problems in the test suite and >movei finds it in less than one second. > >1 can be also solved for the wrong reasons(movei likes it at depth 2-4 only to >change it's mind at depth 5 and to change it's mind again at depth 6 and I >remember that one of the weakest engines showed Bg4 from the first ply) > >There is no doubt that movei can see clear tactics at depth 7 and the score is >+3 after only 1.38 seconds on p850. > >The main problem with 1 is that d4 is another way to win but I expect all >engines to see a better score for Bg4. > >24 is not very easy like 1 but it is also not hard to solve. > >103 seconds and depth 10 on p850 is enough to see fail high on Nxh7. > >The hard problem is 28 and I believe that programs usually solve it for >positional reasons. I now see that you changed 28 to have 2 solutions so by your definition it solved it. I checked latest movei and it solved all the positions in 250 seconds per move except one of the hard positions. I may send you latest movei together with the results if you are intested in it. (you only asked me in email if it is UCI and I answered that it is not. Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.