Author: Christopher A. Morgan
Date: 22:08:26 12/12/02
50 Test Positions, 15 Engines - Results, Comparisons Recently, Gian-Carlo Pascutto, I think it was, posted about 150 position problems, together with solutions, each with an identifying number of ECM.xxx(x). I took the first 51, discarded one, and ran time to solve tests on 15 different chess engines for each of the 50 positions in the Fritz 7 GUI Tools –> Analysis –> Process Test Set window. Below are the results of the tests. The problems in FEN, as previously posted, follow the results. Some details: For each of the problems I confirmed the solution by letting a couple of engines run individually in infinite analysis mode for 5-10 minutes. The one problem discarded had three different solutions by four different engines. For four of the problems, numbers 14, 28, 29 and 38 I listed the text move as the solution together with a variation. In one case Nimzo 8 was only engine of four I tested with that came up with the text solution. The other three engines agreed on a different solution which became the variation. For the other three positions, multiple engines agreed on a different solution which became the variation. My goal was to have the majority of the engines solve every problem, so it would be a test of how quickly a particular engine solved a problem compared to all other engines in average speed of finding solutions to all problems, rather than running for ten minutes (maximum allowed time per position) and not finding a solution. Hardware: Athlon 750, 384MB RAM, 144MB RAM hashtables, except for Chess Tiger 14, and Gambit Tiger 2 which, apparently, only allow a maximum of a 96MB hashtable. The times given should only be looked at in relative terms, that is relative to the other engines. Faster processors will get much faster times, but I would expect that the relative percentage differences in average speed should remain constant among different processors running the same problems with the given solutions. Problems 1, 24, 31, 33, and 41 took the most time for most engines, and a few were not able to find a solution for some of these in the maximum ten minutes allowed. I used the default of looking forward one additional ply after a solution was found. Times to solve problems varied tremendously by problem, and by engine. The fastest engine overall, Nimzo 8, was easily beaten in some problems, for instance. Some examples: Problem 31, Gambit Tiger 2 - 30 seconds 11 ply, Nimzo 8 - 144 seconds 12 ply, and Hiarcs 8 - 353 seconds 11 ply; Problem 41, Nimzo 8 - 22 secs. 9 ply, Ruffian 1.0.1 - 112 seconds 11 ply, and Chess Tiger 14 - 384 seconds 14 ply. Since Nimzo was far and away the fastest problem solver among the 15 engines, does this mean Nimzo is the fastest problem solver among the engines tested? I think the mathematicians here would say that for these 50 problems Nimzo is the fastest, but that we could not extrapolate that further without a great deal of additional testing. Nevertheless, I do find it interesting that Nimzo 8, #15 in the SSDF December 5, 2002 list of 128MB K6-2 450 MHz machines (#32 overall), kicks butt on these 50 problems, easily beating Deep Fritz, #2 on the same list (#15 overall), and Fritz 7 which is #2 overall (F7 engine used is 7.0.0.8). I also found it interesting that Ruffian 1.0.1, a free UCI engine came in number 2 beating a number of commercial engines. The ChessBase file of these 50 problems is available on request from me in .cbv format. Each problem has an automatically generated commentary for each of the 15 engines listing it’s time to solve the problem, and the depth in plies required. If you click on the previous results button in the F7 GUI it will list in order from fastest to slowest the results listed below. If you then click on clip file, you can paste all results into an Excel file, and see all 750 individual results by player and problem number. You have to expand all column widths to see the numbers, but it is formatted properly for Excel. Results in average time in seconds to solve all problems. All 50 problems solved by an engine unless otherwise indicated. 1. Nimzo 8 - 9.6 secs. 2. Ruffian 1.0.1 - 16.7 secs. 3. Fritz 6 - 17.9 secs. 4. Fritz 7 - 19.0 secs. 5. Deep Fritz - 21.3 secs. 6. Gambit Tiger 2 - 22.6 secs. 7. Junior 7 - 28.6 secs. 8. Hiarcs 8 - 28.8 secs. 9. Goliath Light 1.5 - 29.4 secs. 10. Yace 0.99.56 - 49/50 problems solved, 22.5 secs. for solved problems, 34.1 secs. overall 11. Shredder 6.02 - 49/50 problems solved, 24.9 secs. for solved problems, 36.4 secs. overall 12. Chess Tiger 14 - 49/50 problems solved, 27.2 secs. for solved problems, 38.7 secs. overall 13. Gandalf 4.32g - 49/50 problems solved, 31.9 secs. for solved problems, 43.3 secs. overall 14. Pharaon 2.62 - 49/50 problems solved, 52.0 secs. for solved problems, 63.0 secs. overall 15. Crafty 19.01 - 46/50 problems solved, 37.1 secs. for solved problems, 82.2 secs. overall The Position Problems: 1> r1b1r1k1/p1p3pp/2p2n2/2bp4/5P2/3BBQPq/PPPK3P/R4N1R b - - bm Bg4; id "ECM.946"; 2> 1r5r/3b1pk1/3p1np1/p1qPp3/p1N1PbP1/2P2PN1/1PB1Q1K1/R3R3 b - - bm Nxg4; id>"ECM.949"; 3> 2k4r/1pp2ppp/p1p1bn2/4N3/1q1rP3/2N1Q3/PPP2PPP/R4RK1 w - - bm Nd5; id "ECM.952"; 4> rn3rk1/4bppp/1q2p3/p2pP3/8/1PN2B1P/P4PP1/2RQ1RK1 w - - bm Bxd5; id "ECM.961"; 5>2r2bk1/4qp2/3n2p1/2R1p1Np/2p1N3/r6P/1Q3PP1/3R2K1 w - - bm Rxc8; id "ECM.973"; 6>r1b1rnk1/1p3pb1/1qpp2p1/2n5/2PNP2p/1PN3PP/3R1PBK/BR1Q4 b - - bm Rxa1; id >"ECM.977"; 7>r1b2rk1/pp2b3/2pn1n1p/3pNppq/3P4/BP1N2P1/P3PPBP/R1Q2RK1 w - - bm Nxc6; id >"ECM.978"; 8> 3rkb1r/1p3p2/p1n1p3/q5pp/2PpP3/1P4P1/P1Q1BPKP/R2N3R b k - bm d3; id "ECM.986"; 9> 5rk1/1r1qbnnp/R2p2p1/1p1Pp3/1Pp1P1N1/2P1B1NP/5QP1/5R1K w - - bm Qxf7+; id >"ECM.987"; 10> 5r2/1b2k2p/3bp3/5PP1/Bn1Q3P/1N6/1Pr5/1K3R2 b - - bm Rxb2+; id "ECM.993"; 11> r4rk1/1b3ppp/p2q1n2/1p2N3/3P4/1B6/P1Q2PPP/2R1R1K1 w - - bm Nxf7; id "ECM.999"; 12> 3r1rk1/1b2bppp/2q1pn2/pp4Q1/3B1P2/PBN5/1PP3PP/R2R3K b - - bm Rxd4; id >"ECM.1004"; 13> 3r4/p2nrpkp/2B1p3/2P2pP1/3R3Q/q7/6PP/6RK w - - bm g6; id "ECM.1007"; 14> b1n1r1k1/2q2p1p/p4Pp1/1pBP4/4B1P1/2P1Q3/P4K1P/4R3 w - - bm Qh6 (variation Kg2); id "ECM.1009"; 15> r1bqnrk1/pp2ppb1/1np3pp/4P1N1/5P2/2NBB3/PPP3PP/R2Q1RK1 w - - bm Nh7; id >"ECM.1011"; 16> 4rrk1/1bp2ppp/p1q2b1B/1pn2B2/4N1Q1/2P4P/PP3PP1/3RR1K1 w - - bm Nxc5; id >"ECM.1016"; 17> 5r1k/pb2r1bp/1p2B1p1/n7/4qNQ1/4B3/P4PPP/2RR2K1 w - - bm Rd4; id "ECM.1018"; 18> 1r3rk1/5pp1/1n2qn1p/1p1pPNb1/2pP2QP/2P5/1PB3P1/R1B1R1K1 w - - bm exf6; id >"ECM.1020"; 19> 2r1rbk1/1b3ppp/pp6/2q1pNP1/Pn1RP3/2N5/1PP2QBP/5R1K w - - bm Nh6+; id "ECM.1025"; 20> r4k2/q4npp/P2P4/2p1R3/2N5/6PP/Q5K1/8 w - - bm Re7; id "ECM.1028"; 21> rr4k1/2p1ppb1/3pbnpp/q1pP4/4P3/1PNNBP1P/P1KR2P1/2Q4R b - - bm Qxc3+; id >"ECM.1038"; 22> 5rk1/p1pb2pp/2p5/3p3q/2P3n1/1Q4BN/PP1Np1KP/R3R3 b - - bm Qxh3+; id "ECM.1041"; >23> r1k4r/ppp1b3/5Npp/4pb1Q/8/1B2B3/PqP2PPP/2RR2K1 w - - bm Qxf5+; id "ECM.1045"; >24> r1bn1rk1/pp3p1p/6p1/2bR2N1/2B2B2/q1P1P3/2Q2PPP/4K2R w K - bm Nxh7; id >"ECM.1053"; >25> 2b2r2/1p2q1k1/r3pppp/4n2P/p3N3/1B4Q1/PPP3P1/3R1R1K w - - bm Rxf6; id "ECM.1057"; 26> rbbq1rk1/p3nppp/1p2p3/8/1B1pN3/P2B4/1P3PPP/2RQ1RK1 w - - bm Nf6+; id "ECM.1065"; 27> rn2qbr1/2p4k/p2p1nb1/1p1Pp2p/2P4P/2NBBPN1/PP1Q4/2KR2R1 w - - bm Nxh5; id >"ECM.1066"; 28> r1bq1rk1/pp2bp1p/2p2np1/3p2B1/3P4/2NQ2N1/PPP2PPP/4RRK1 w - - bm Rxe7 (variation f4); id >"ECM.1068"; 29> 2k3r1/pp2rp2/1np5/2Np1p2/P2P3p/1R2P1Pq/2Q2P1P/1R4K1 w - - bm Nxb7 (variation a5); id >"ECM.1073"; 30> 2rr1bk1/5p1p/pPN2np1/3Bp3/2Q1n3/1P2B1Pq/P3PP2/R2R2K1 b - - bm Rxd5; id >"ECM.1076"; 31> b3r3/q2B2k1/3Q2p1/1p5p/3pP3/5P2/1p4PP/5RK1 b - - bm d3+; id "ECM.1080"; 32> 2r3k1/pp2n3/6pQ/4q3/8/2P1p1P1/P5BP/3R2K1 w - - bm Bd5+; id "ECM.1082"; 33> 8/p3q1kp/1p2Pnp1/3pQ3/2pP4/1nP3N1/1B4PP/6K1 w - - bm Ba3; id "ECM.1084"; 34> r2b1rk1/5qp1/2p2P2/2p3Pp/p6P/2Q5/PPP5/2KR2R1 w - - bm g6; id "ECM.1085"; 35> 4r3/Q2bppkp/3p2p1/3Nn3/4P3/6PP/1qr2PB1/R3R1K1 b - - bm Ra8; id "ECM.1093"; 36> 8/p5Q1/2ppq2p/3n1ppk/3B4/2P2P1P/P5P1/6K1 w - - bm g4+; id "ECM.1096"; 37> 1r2k2r/3q2pp/p3pp2/P7/2P1Q3/8/1nB3PP/1R3R1K w k - bm Ba4; id "ECM.1098"; 38> 3R4/1p2kp2/p1b1nN2/6p1/8/6BP/2r1qPPK/Q7 w - - bm Rd7+ (variation Rb8); id "ECM.1100"; 39> 1q6/2rk1p2/p1n1b1rN/3p2p1/8/3Q4/PP1B1PPP/2R1R1K1 w - - bm Nxf7; id "ECM.1114"; 40> 3r3k/p1Q3bp/2p3p1/8/2P1BRn1/1P4Pb/P1N1q2P/6BK b - - bm Nf2+; id "ECM.1122"; 41> r1b4r/pp2ppk1/2np1np1/3N3p/2B1P3/q3BP2/P1PQ2PP/1R3RK1 w - - bm Nxe7; id >"ECM.1125"; 42> 5rk1/Q2nqppp/5n2/2p5/8/1B2PP1P/Pr1B1P2/R3K2R b KQ - bm Ne4; id "ECM.1127"; 43> r2r1bk1/3qp2p/3pp1p1/p2n2N1/2N3Q1/BP4P1/P4PP1/2R3K1 w - - bm Nb6; id "ECM.1131"; 44> 1r3rk1/p4pbp/q1p2pp1/3p1b2/1n1P1P1B/PB3N2/1PPQR1PP/2KR4 b - - bm Bxc2; id >"ECM.1132"; 45> 1rbr2k1/4qp2/p1n2bpp/1pp1p1N1/4P3/2P1BQ1P/PPB2PP1/3RR1K1 w - - bm Bxc5; id >"ECM.1133"; NOT USED - Engines don’t agree on best move: >2r2rk1/1p1q1ppp/1p3n2/3p1N2/4n3/1N3Q2/PPP2PPP/R2R2K1 w - - bm Rxd5; id >"ECM.1134"; 46> 2r1r1k1/pp3pp1/3p3B/P2P1P2/2nb4/7R/1q3PQP/1B3R1K w - - bm Rb3; id "ECM.1147"; 47> 1k1r2r1/pp3p1p/B2q1n2/8/3Pb1p1/2Q5/PR3PPP/2B1R1K1 w - - bm Bf4; id "ECM.1150"; 48> k3r3/pR5p/PppR1p1p/4nP2/1PP5/8/4B1rP/2K5 w - - bm Re7; id "ECM.1151"; 49> 1q3rk1/r5b1/pNp1bn1p/2Pp1n2/1P2p3/1N2P2P/PBQKB3/3R3R b - - bm Nxe3; id >"ECM.1152"; 50> r5k1/5p1p/bq1p2p1/r2Pp3/1p2N1P1/1P3P2/1KPQ3P/3R3R b - - bm Ra2+; id "ECM.1160"
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.