Author: Don Dailey
Date: 17:07:30 01/19/98
Go up one level in this thread
Hi Amir, I don't like the flaky ones at all, (unlike you) but I am thinking it's ok to include the easy problems. If the set can properly be run until a solution is found and then quit, the easy problems do not consume much time and I think that was the big concern. Also I like to run at different times. For instance we may all do a 2 minute run to compare but I may want 5 second runs for quick tests depending on what I'm trying to do. The flaky problems makes this quite useless for me because I can already see I will get a LOT of problems in 5 seconds and lose them later (even if I run for full time method.) I agree with you that a big set is useful and even the flaky ones are also useful but not for benchmarking unless you are willing to except a margin of error (which in this set may be bigger than usual because the author included a lot of normal moves and called them brilliant.) It comes down to what you want to use the set for. If we want it to see how well our program can see tactics then seeing the right move on the 2nd iteration is not very useful is it? - Don On January 19, 1998 at 18:04:29, Amir Ban wrote: >On January 19, 1998 at 13:30:39, Robert Hyatt wrote: > >>On January 19, 1998 at 03:39:23, Don Dailey wrote: >> > >> >>the only thing I want to see is a suite where each position has a clear >>solution, whether it be hard or easy, but where the solution move is not >>something that might show up "just because". IE the solution should be >>a tactical solution that requires a move that wouldn't normally be >>played, >>so we can compare apples to apples. The position Bruce mentioned from >>the >>BK test is but one example. Many people have, over the years, reported >>WAC results where they simply found the best move positionally without >>having any idea that it ends up winning material or mating. I'd like to >>see our suite produce unambiguous results... >> >> >>I'd like to suggest the following way to get this started: >> >>lets take the first 100 positions to begin with. If we all agree that >>10 seconds is too easy, that's fine by me. If we'd rather look at them >>more carefully and eliminate the 3-4 ply solutions instead, it doesn't >>matter. But first, let's eliminate the easy ones. After we agree on >>those, then lets eliminate the ambiguous ones, or the ones with too many >>solutions, or the ones that we can't confirm that the solution is really >>best, etc... >> >>I will tally the results. I'd like to see Bruce do the same. Then we >>can compare notes to be sure neither missed a comment by someone. When >>we >>reduce the first 100 to N, then we move to the next 100. 100 seems >>managable, >>without producing huge posts. > > >I don't object to your plan, but I have a slightly different target in >mind. I see more advantages than drawbacks in having a large suite. What >I don't want is to count one less when I should be counting one more >because the solution is not correct or there are others. I don't mind >the "flakes", and I think the "easy" ones have a place because this >means the suite has sensitivity to running at very short times. > >By all means go ahead with your plans, but my first priority is to get >the bugs out of the suite. > >Amir
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.