Author: Don Dailey
Date: 16:44:02 05/02/98
Go up one level in this thread
On May 02, 1998 at 13:40:31, Howard Exner wrote:
>On May 02, 1998 at 12:51:16, Bruce Moreland wrote:
>
>>
>>On May 02, 1998 at 11:34:38, Howard Exner wrote:
>>
>>>On May 01, 1998 at 14:38:14, Don Dailey wrote:
>>
>>>>B) Solution should not be an obvious move that any program has
>>>> some chance of playing even if it doesn't really understand the
>>>> idea.
>>>
>>>On this point B I recognized some key moves as being played to achieve a
>>>draw
>>>rather than seeing deeper that they are winning. Botvinnik's Ba3 against
>>>Capablanca comes to mind. Would this be the kind of example you are
>>>referring
>>>to?
>>
>>It'd be awful to throw out a good position like this just because after
>>X seconds some programs score it as 0 rather than +3, right?
>
>Definitely keep this one of course, as Don suggests too(ie: as Don said
>keep it if it doesn't really understand the idea). Why did you think
>I wanted to toss it?
>>
>>bruce
I haven't looked at the position. But let me explain this criteria
a little better:
Ideally, a really clean set would have these characteristics (in my
opinion):
A) The key or key(s) will have a unique game theoretic value to the
other moves.
B) The key or key(s) are unlikely to get played by accident.
CASE A means one of two things:
1. All moves lead to a loss except the key move(s) which draw or win.
or
2. All moves lead to a draw except the key move(s) which give a win.
CASE B is illustrated by this example:
If "O-O" is a move that leads to the win of a piece in 10 ply, and
yet is such a natural move that every program plays it on 1 ply,
do you want to include it? I don't want to include it but others
might.
But I think your comment was directed to case A. If the key move can
force at least a draw and all other moves are a loss, then it's
a good problem. If the key move turns out also be a win, that's
ok too, the right move was found for the sound reason that it
avoided a loss.
We had a discussion a while back on these "flaky" key moves.
Again, I define flaky as a natural move that is also key.
The problem in my
opinion is that any weak program can get them right completely
by accident. We could not meaningfully compare our results to
each other and trust them. Bob Hyatt and others posted that
they did not want problems that weak programs could do well
on by accident. He also gave arguments why we should keep them
so I'm not clear on how he stood. But there was not a strong
consensous on this either way so I was confused.
The technique of waiting to see if your program keeps the solution
until some arbitrary time limit is not of much help unfortunately.
The assumption is that if your program doesn't see the move
for the right reason, a different move will happen to be in the
the display at the end of the time limit. Since this does happen
occasionally, the accuracy of your scoring is slightly improved,
but at the cost of much more testing time. I personally do not
like this technique and cringe at its imprecision and randomness.
If I were testing a positional problem set I might use the
technique for the simple reason that it's better than not doing
it. But with a tactical set why not make it unambiguous?
It makes the most sense to just concentrate on having a CORRECT
set. Once we have this, anyone who want to help me get a cleaner
subset out of this can and we'll all have two nice sets to play with.
- Don
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.