Author: Howard Exner
Date: 10:57:32 01/11/99
Go up one level in this thread
On January 10, 1999 at 17:05:44, STEPHEN A. BOAK wrote: >Howard Exner has an interesting article entitled 'Rebel 10 in the Endgame', >posted on here on CCRC. His writeup indicates he gave 7 full minutes for >analysis of each position. > >My subject is only his first position: > >EXNER EPD Suite, Position #01 >5k2/8/7P/8/8/8/K6P/1B6 w - - id END01; bm Bh7; > >Howard shows the following test results for various Rebel software versions: > >Endgame Test Suite - Rebel 8,9 and 10 > >Machine - AMD K-233 with 60 MB Hash running in Dos > >8-D is Rebel 8 default | 9-D is Rebel 9 default | 10-NC is Rebel 10 with >Combination Off 10-ST4A is Rebel 10 with Combination off, AG=Strong, Selection = >4 and Play Style = Active > > X denotes not solved in under 7 minutes | am means "avoid move" > > POS Move > 1 8-D 9-D 10-NC 10-ST4A 1 Bh7 > 0:52 0:48 1:03 1:20 > >I analyzed the same position for approximately 12 hours, using ECTool and >special Rebel 10 analysis engine for ECTool. This software combination only >allows up to 28MB maximum (which I used). I used a Pentium 200 (not MMX) >machine. > >Here are my analysis results: > >Rebel Engine for ECTool. (c) Ed Schröder > >Engine version : REBEL 10 >Hash table size : 28 Mb >Analysis mode : Analyzing next move >Refresh interval : 1000 ms > >Game begin > >00:01 08.01 5.99 Bb1-h7 Kf8-f7 Ka2-b2 Kf7-f8 Kb2-c3 Kf8-e7 Bh7-e4 >00:02 10.00 5.29 Bb1-h7 Kf8-e7 Ka2-b3 Ke7-f7 Kb3-c4 Kf7-f8 Bh7-e4 >00:02 10.01 5.36 Bb1-e4 Kf8-g8 Ka2-b3 Kg8-h8 Kb3-c4 Kh8-g8 Kc4-d4 >00:03 11.01 6.23 Bb1-h7 Kf8-e7 Ka2-b3 Ke7-f6 h2-h4 Kf6-f7 h4-h5 >00:06 12.00 5.36 Bb1-h7 Kf8-e7 Ka2-b2 Ke7-f6 Bh7-e4 Kf6-f7 Kb2-c3 >00:06 12.01 5.42 Bb1-e4 Kf8-g8 Ka2-b3 Kg8-h8 Kb3-c4 Kh8-g8 h2-h4 >00:09 13.00 5.42 Bb1-e4 Kf8-g8 Ka2-b3 Kg8-h8 Kb3-c4 Kh8-g8 h2-h4 >00:13 14.00 5.43 Bb1-e4 Kf8-g8 Ka2-b3 Kg8-h8 Kb3-c4 Kh8-g8 Kc4-d4 >00:13 14.01 5.43 Bb1-h7 >00:16 14.01 6.37 Bb1-h7 Kf8-e7 h2-h4 Ke7-f6 Ka2-b3 Kf6-f7 Kb3-c4 >00:24 15.00 5.45 Bb1-h7 Kf8-e7 h2-h4 Ke7-f6 Ka2-b3 Kf6-f7 Kb3-c3 >00:26 15.01 5.50 Bb1-e4 Kf8-g8 Ka2-b3 Kg8-h8 h2-h4 Kh8-g8 Kb3-c4 >00:37 16.01 5.46 Bb1-h7 >00:39 16.01 6.41 Bb1-h7 Kf8-e7 Ka2-b3 Ke7-f7 h2-h4 Kf7-f6 Kb3-c4 >00:54 17.00 6.38 Bb1-h7 Kf8-f7 Ka2-b3 Kf7-e7 Kb3-c4 Ke7-f7 h2-h4 >01:13 18.00 5.50 Bb1-h7 Kf8-f7 Ka2-b3 Kf7-e7 Bh7-e4 Ke7-f7 Kb3-c4 >01:19 18.02 5.51 h2-h4 Kf8-g8 Ka2-b2 Kg8-h8 Kb2-c3 Kh8-g8 Kc3-d4 >01:33 19.00 5.68 h2-h4 Kf8-g8 Ka2-b2 Kg8-h8 Kb2-c3 Kh8-g8 Kc3-d4 >01:39 19.01 5.68 Bb1-h7 >01:50 19.01 6.48 Bb1-h7 Kf8-f7 Ka2-b3 Kf7-e7 Kb3-c4 Ke7-f7 h2-h4 >02:35 20.00 6.72 Bb1-h7 Kf8-f7 Ka2-b3 Kf7-e7 Kb3-c4 Ke7-f7 h2-h4 >03:20 21.00 6.72 Bb1-h7 Kf8-f7 h2-h4 Kf7-f8 Ka2-b3 Kf8-f7 Kb3-c4 >46:47 22.00 13.42 Bb1-h7 Kf8-f7 h2-h4 Kf7-f6 Ka2-b3 Kf6-f7 Kb3-c4 >49:57 23.00 13.56 Bb1-h7 Kf8-f7 h2-h4 Kf7-f6 Ka2-b3 Kf6-f7 Kb3-c4 > >Note that the times shown by ECTool only are minutes and seconds. After a full >hour, the minutes may drop from 59 min to 1 min. Therefore I can not tell >exactly how many hours (plus the shown minutes and seconds) elapsed for the last >two or three analysis lines. I quit analyzing after approx 12 hours, and the >last two lines had already finished, as shown above. > >My question for Howard Exner is this--what constitutes a solved position in his >endgame test suite test of Rebel versions? > >The last shown best move at the end of 7 minutes of analysis? > >The first time posted, for the last shown best move at the end of 7 minutes, >after which the added lines did not change the best move? I'm taking some guidance on this suite from previous popular suites like the BT series or the Louguet. In these suites the times are recorded when a program first "plays the move" from its PV score and sticks to that move up until the fixed time. There is a downside to this method as to whether it will really predict what a program will play in an actual game. During an actual game a program may find a move immediately as it could have found it during the opponents thinking time. Or else if a program correctly predicts an opponents move then it has a head start on the solution time. Another factor is that programs today are improved in the way they manage their clocks. Sometimes in an actual game programs can take quite awhile before they move. To me this is quite impressive how the programmers are managing this. > >Do you use the PV score to determine if a solution is 'understood' by the >program? This is an important point you are making when a program is given a position to "solve". What constitutes solved? Sometimes the correct move is played because of a minute eval over another position. In the end I just adopt what Don Dailey discribed here once as the computer as a "black box". Whatever is going on inside this black box it decides to churn out a move at some point and we have to live with its choice, not really knowing if it "understands" the poition. I tried in this suite to gather/create positions that were critical, meaning that only one correct move is allowed to gain the win or to secure a draw. (or to avoid a single move that would toss away a draw or win). > >In my analysis results, for example, ECTool/Rebel 10 waffled between a total of >3 different moves, over 21 ply levels, showing the score in the 5 to 7 point >range (White has B+2P; Black a King only). Even on ply 19.00, the PV showed h4 >as the best move (incorrect solution). On ply 22, the program finally >calculated a PV score of 13+ for White, indicating it finally saw a winning pawn >promotion. > >Howard tested the ability of Rebel software versions to find endgame solutions >for his EPD suite of 30 positions, during typical real life game analysis times >(he used 7 full min per position). For this purpose his test provides useful >feedback. > >A test of limited time duration will not always allow a program to finalize a >best move (waffling may still occur, as analysis continues). I've seen this on some test suites too. Sometimes a program was given credit for playing the correct move on slower hardware. But when the same program is run on faster hardware (or given more time for analysis) it plays a move that is no longer tagged as the correct response. >However, in a constructed test position where there is a specific solution, >until the program sees the 'gain' from the best line of play (either mate or >substantial material increase), it may continue to waffle, depending on how much >time it has in an actual game to make a decision. It makes me wonder if the >program really 'has' or 'would' solve the problem in actual play. Perhaps, for >example, a program might waffle back and forth, each turn to play, taking up to >7 minutes to find best moves but never fully understanding them enough to drive >the position to a complete solution. > >This is food for thought--any others with ideas on the subject? > >--Steve
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.