Author: Mark Young
Date: 11:23:54 06/12/99
Go up one level in this thread
On June 12, 1999 at 13:31:20, James Robertson wrote: >Whenever I send out a new version of my program, I usually do a lot of testing >vs. other programs to see how good it is. Usually I get some really cool result, >and can't wait for everybody to see how good my new version is. > >But then the test results start coming back: "Your program is very good; it only >lost 300-1 to GNUchess", or "It had a mishap in it's match against Faile, but >other than that it's quite good". > >It's not fair!!! How come most people send back results far worse than the ones >I get on my computer!? I am so careful to check that the hash sizes are the >same, the CPU use is the same..... > >Does anybody else have this experience? > >James I see examples of this type with the commercial programs on CCC being tested by members. Thats why it is importent to not only have the results, but for the tester to show the games with the eval and think times if possible, and with the computer and program settings. This way a result like 300 to 1 can be checked, and see if the games can be reproduced along with the book lines. My trust for many testers is 0%, I have seen some strange test results that I could not in a 100 years reproduce. Not to say they all lie about the results, but some results you see, the only conclusion to come to is they played a 40 game match where program X wins big, then post the match results as a 10 game match where program y wins 10-0, and they have the games to prove it. :)
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.