Author: Vincent Diepeveen
Date: 16:57:10 12/17/02
Go up one level in this thread
On December 17, 2002 at 19:44:03, Dann Corbit wrote: >On December 17, 2002 at 19:36:09, Vincent Diepeveen wrote: > >>On December 17, 2002 at 19:10:42, Dann Corbit wrote: >> >>Dann, if your goal is to measure what works better for your program >>in real life, then of course playing games against other >>programs than your own thing proofs most. >> >>But what is most important when you play another program? >>Right: the quality of the move within the time you have. >> >>Of course that testing is very time consuming. >> >>So if you want to start proving, before doing the >>game playing test, that it also is losslessly solving >>more positions for example, then the only way to do >>that is in a fixed time. >> >>It isn't better then to solve 10 positions in 0:01 and 10 not >>versus another algorithm solving 12 positions in 0:10 and 8 not. >> >>What matters is whether you find it within the time control you play. >> >>The human time control is like 3 minutes a move, but all the big mistakes >>programs make are basically first move out of book. I had to my big >>shame again 2 big examples of that last tournament i played (90 0 level) >> >>I do not know about your program but in DIEP that means something like >>between 5 and 15 minutes. 5 minutes at least at 90 0 and like 15 minutes >>at 3 minutes a move like i play in paderborn ipccc2003 (40 in 2 + 1 all). > >The time control chosen will be a function of the strength of both the program >and the hardware together with the total test effort needed. > >If the goal is to test 100 programs on 20 test sets of average 50 positions, >nobody can do it at 10 minutes per position. with diep i can do this easily. i skip positions which are solved. that code is not so easy to figure out but it is trivial to make from design viewpoint. the vaste majority of testset positions you always solve very quickly. that saves days of testing time i bet :) >>It is incorrect to compare number of nodes for an obvious reason. >> >>suppose i want to show my parallel algorithm works fine. If i make >>some crappy algorithm which basically takes care just 1 processor >>searches then i can conclude my parallel algorithm is performing >>excellent. >> >>The only decent way to measure my speedup is by looking at the >>search times. > >Assuming that the solution is found (of course). But I agree here. If the >nodes are less but the time to solution is more, then you have created a smarter >idiot. ;-) > >>Of course the hard thing with parallellism is that some of the >>researchers (and i do NOT mean Hyatt here) first slowed down >>their program 30 times in order to then get a better speedup. > >Even though I have seen some strange algorithms employed in parallel analysis >(and some of them are absolutely wrong for sure) it seems that there still may >be interesting ideas imbedded in the work. Worth a gedankenexperiment at least. > >>That's another issue which is hard to overcome. >> >>Some zugzwang articles are very convincing there how to NOT do it, >>just as bad is Cilkchess which i could see with my own eyes getting >>5000 nodes a second at 1 cpu and a version not using cilk running >>at a single processor getting far over 100k nps :) > >Ouch. Seems a little optimization might be handy there. That seems stupendous >overhead. > >>Testing will never be easy as long as researchers want to present something >>crappy as good, because only a crappy way of testing then can conclude >>it is good. >> >>Another method to show your thing works great is by not comparing with >>a good other algorithm. >> >>Suppose i want to conclude my algorithm works great, why not compare it >>with minimax? >> >>That's of course a great way to show my thing works good. > >You bring up a good point that is often overlooked in research. What is the >control? I have seen multimillion dollar experiments conducted with no control >at all. The design of the control is as important as the design of the >experiment.
This page took 0.01 seconds to execute
Last modified: Thu, 07 Jul 11 08:48:38 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.