Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: misleading Pet test suite

Author: José Carlos

Date: 04:13:14 12/07/03

Go up one level in this thread


On December 07, 2003 at 06:43:40, Uri Blass wrote:

>I tested my engine in the pet test suite and I found that it failed to solve 25
>that was mentioned as one of the easy positions by Thomas Mayer(it solved it but
>unforntuanately changes its mind later)
>
>see http://f11.parsimony.net/forum16635/messages/49405.htm
>
>I decided to test Yace padderborn and it solves it in less than 1 seconds but
>the score if clearly bad for white and the main line is wrong at least in the
>first iterations.
>
>Yace does not see the draw and I think that a scoring system that gives only
>points for correct move without looking at the main line and the evaluation is
>clearly wrong and probably a lot of engines that solved 25 fail to find all the
>right moves that are given in
>http://homepages.caverock.net.nz/~peter/eg_test/pet025.htm and may lose the game
>if they are white inspite of first correct move(Yace seems to find more and more
>correct moves in its main line when I give it more time so it probably can save
>the game)
>
>
>New game
>[D]4K3/2k1Bp1N/6p1/5PP1/8/7p/b7/8 w - - 0 1
>
>Analysis by Yace Paderborn:
>
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 1   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 1   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 1   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 fxg6 2.Bd8+ Kd6 3.Nf6
>  +-  (1.49)   Depth: 3   00:00:00
>1.fxg6 h2 2.g7 f6 3.g8Q Bxg8
>  ±  (1.26)   Depth: 4   00:00:00
>1.fxg6 h2 2.g7 f6 3.g8Q Bxg8
>  ±  (1.26)   Depth: 4   00:00:00
>1.fxg6 fxg6 2.Bf6 Kd6 3.Nf8 h2
>  ±  (1.03)   Depth: 5   00:00:00
>1.fxg6 fxg6 2.Bf6 Kd6 3.Nf8 h2
>  ±  (1.03)   Depth: 5   00:00:00
>1.fxg6 fxg6 2.Nf6 h2 3.Ne4 h1Q 4.Bf6
>  ²  (0.63)   Depth: 6/14   00:00:00  15kN
>1.fxg6 h2 2.g7 f6 3.g8N h1Q 4.gxf6
>  ³  (-0.37)   Depth: 6/14   00:00:00  24kN
>1.fxg6 h2 2.gxf7 Bxf7+ 3.Kxf7 h1Q 4.g6 Qd5+ 5.Kf6 Kd7
>  µ  (-1.17)   Depth: 6/17   00:00:00  40kN
>1.Bf6 gxf5
>  µ  (-0.84)   Depth: 6/17   00:00:00  49kN
>1.Bf6 gxf5 2.Be5+ Kc6 3.Nf6 Bc4 4.Bh2
>  ±  (0.93)   Depth: 6/17   00:00:00  52kN
>1.Bf6 gxf5 2.Be5+ Kc6 3.Nf6 Bc4 4.Bh2
>  ±  (0.93)   Depth: 6/19   00:00:00  55kN
>1.Bf6 Kd6 2.Kf8 h2 3.Bb2 h1Q 4.fxg6
>  ²  (0.53)   Depth: 7/19   00:00:00  68kN
>1.Bf6 Kd6 2.Kf8 h2 3.Bb2 h1Q 4.fxg6
>  ³  (-0.47)   Depth: 7/19   00:00:00  72kN
>1.Bf6 Kd6 2.fxg6 h2 3.g7 h1Q 4.g8Q Kc6
>  ³  (-0.47)   Depth: 7/19   00:00:00  83kN
>1.Bf6 Kd6 2.fxg6 h2 3.g7 h1Q 4.g8Q Kc6
>  ³  (-0.47)   Depth: 7/19   00:00:00  119kN
>1.Bf6 Kd6 2.fxg6 fxg6 3.Kf8 h2 4.Ke8 h1Q 5.Kf8
>  µ  (-0.87)   Depth: 8/19   00:00:01  137kN
>1.Bf6 Kd6 2.fxg6 fxg6
>  µ  (-0.87)   Depth: 8/19   00:00:01  143kN
>1.Bf6 Kd6 2.fxg6 fxg6
>  µ  (-0.87)   Depth: 8/19   00:00:01  208kN
>1.Bf6 Kd6 2.fxg6 fxg6 3.Be7+ Kc6 4.Bf6 h2 5.Be5 h1Q 6.Bh2
>  µ  (-1.27)   Depth: 9/19   00:00:01  245kN
>1.Bf6 Kd6 2.fxg6 fxg6 3.Be7+ Kc6 4.Bf6 h2 5.Be5 h1Q 6.Bh2
>  -+  (-2.27)   Depth: 9/19   00:00:01  285kN
>1.Bf6 Kd6 2.fxg6 fxg6 3.Be7+ Kc6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2
>  -+  (-2.61)   Depth: 9/21   00:00:01  351kN
>1.Bf6 Kd6 2.fxg6 fxg6 3.Be7+ Kc6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2
>  -+  (-2.61)   Depth: 9/25   00:00:02  812kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Ke7 h1Q 8.Nf6
>  -+  (-2.73)   Depth: 10/25   00:00:02  1001kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Ke7 h1Q 8.Nf6
>  -+  (-2.73)   Depth: 10/25   00:00:03  1371kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Nh5 h1Q 7.Ng3 Qh7 8.Nf5+
>gxf5 9.g6
>  -+  (-3.13)   Depth: 11/27   00:00:04  1616kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.Bd6 Kxd6 4.Nf6 h2 5.Ne4+ Ke5 6.Ng3 gxf5 7.Nh1 f4
>  -+  (-3.47)   Depth: 11/29   00:00:04  1969kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.Bd6 Kxd6 4.Nf6 h2 5.Ne4+ Ke5 6.Ng3 gxf5 7.Nh1 f4
>  -+  (-3.47)   Depth: 11/29   00:00:07  3019kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Ng3 Kf4 8.Nh1
>Kxg5
>  -+  (-3.70)   Depth: 12/30   00:00:09  4009kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Ng3 Kf4 8.Nh1
>Kxg5
>  -+  (-3.70)   Depth: 12/31   00:00:13  6085kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2 Kf4 8.Ke7
>Kxg5 9.Nh1
>  -+  (-3.71)   Depth: 13/36   00:00:22  10165kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2 Kf4 8.Ke7
>Kxg5 9.Nh1
>  -+  (-3.71)   Depth: 13/36   00:00:35  16295kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2 Kf4 8.Nh1
>Bd5 9.Nf2 Kxg5 10.Kf8
>  -+  (-3.99)   Depth: 14/36   00:00:52  24389kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2 Kf4 8.Nh1
>Bd5 9.Nf2 Kxg5 10.Kf8
>  -+  (-3.99)   Depth: 14/36   00:01:24  39229kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ng4 h1Q 7.Nh2
>  -+  (-4.39)   Depth: 15/36   00:01:38  45423kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nc3
>  -+  (-5.39)   Depth: 15/36   00:01:50  51152kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2 Kf4 8.Ke7
>Kg3 9.Nh1+ Kg2 10.Kf6 Kxh1
>  -+  (-5.54)   Depth: 15/38   00:02:21  64828kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Ne4+ Ke5 7.Nf2 Kf4 8.Ke7
>Kg3 9.Nh1+ Kg2 10.Kf6 Kxh1
>  -+  (-5.54)   Depth: 15/38   00:03:55  106272kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.fxg6 fxg6 4.Bd6 Kxd6 5.Nf6 h2 6.Kd8 h1Q 7.Nh5
>  -+  (-5.94)   Depth: 16/38   00:04:23  118123kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.f6 h2 4.Kf8 h1Q 5.Kg7 Qh5 6.Kg8 Bc4 7.Bd8 Kd6 8.Be7+ Ke5
>9.Bc5 Qh4 10.Be7
>  -+  (-6.66)   Depth: 16/41   00:05:37  152198kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.f6 h2 4.Kf8 h1Q 5.Kg7 Qh5 6.Kg8 Bc4 7.Bd8 Kd6 8.Be7+ Ke5
>9.Bc5 Qh4 10.Be7
>  -+  (-6.66)   Depth: 16/43   00:10:00  275874kN
>1.Bf6 Kd6 2.Be7+ Kc6 3.f6 h2 4.Kf8 h1Q 5.Kg7 Kd5 6.Bd8 Qh5 7.Kh8 Bc4 8.Bb6 Qh4
>9.Kg8
>  -+  (-6.70)   Depth: 17/45   00:12:45  348980kN
>
>(Blass, Tel-Aviv 07.12.2003)
>
>Uri

  This is a bad test position, IMO. Most programs don't understand the fortress
concept at all, so they're seing everything loses. The bishop moves to try to
stop the pawn just push the loss farther, so the programs soon choose it for the
wrong reason.
  My programs find Bf6 quickly, in less than a second, but they have no idea
what a fortress is.

  José C.



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.