Author: Robert Hyatt
Date: 07:35:12 06/16/98
Go up one level in this thread
On June 15, 1998 at 19:19:55, jonathan Baxter wrote: >On June 15, 1998 at 15:03:27, Robert Hyatt wrote: > >>The point is not nearly so much about the SSDF methods as is is about the >>greediness of chess programmers. Once an experiment is started, and the >>details of the testing methodology become known, the experiment is over. >>That's why drug trials are "double-blind" in the US. *no* one involved >>knows who is getting the real drug and who is getting a placebo. So once >>the testing conditions became known, we started seeing killer books, odd >>behavior (running out of time if first non-book move was bad, so the game >>would be aborted) and other such things. None of which was caused by the >>SSDF... but rather by the programmers themselves... >> >>We've seen cooked test suites, cooked book lines, and a host of such things, >>so *nothing* should be a big surprise any more... > >Cooked books are ok, after all humans do *exactly* the same thing and we don't >object to that. Any program worth its salt should adapt quickly to the cooks and >stop playing into them. So at least in a long enough match the cooks will even >out. sure, but suppose you know that the typical SSDF tester plays maybe 20 games between two programs before he goes on to a new opponent for one of them? And suppose you know there are *many* testers that will run such short matches. If you provide 20 cooks in your book, you get 20 wins. And by the time the other program has learned to avoid the cooks, the match on that machine is over and the learned results don't prevent a 20-0 match result. As I said, if you know the testing methodology, you can figure out ways to turn it into an advantage. It's happened more than once...
This page took 2.74 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.