Author: Stuart Cracraft
Date: 07:29:33 10/02/04
Go up one level in this thread
On October 01, 2004 at 21:51:55, Robert Hyatt wrote: >On October 01, 2004 at 19:40:18, Stuart Cracraft wrote: > >>On October 01, 2004 at 00:04:03, Robert Hyatt wrote: >> >>>On September 29, 2004 at 19:50:15, Stuart Cracraft wrote: >>> >>>>On September 29, 2004 at 17:47:27, martin fierz wrote: >>>> >>>>>On September 29, 2004 at 13:44:50, Stuart Cracraft wrote: >>>>> >>>>>>On September 29, 2004 at 08:22:38, martin fierz wrote: >>>>>> >>>>>>>On September 28, 2004 at 23:49:01, Stuart Cracraft wrote: >>>>>>> >>>>>>>>On September 28, 2004 at 16:29:29, martin fierz wrote: >>>>>>>> >>>>>>>>>On September 28, 2004 at 13:43:48, Stuart Cracraft wrote: >>>>>>>>> >>>>>>>>>>On September 28, 2004 at 08:44:04, martin fierz wrote: >>>>>>>>>> >>>>>>>>>>>On September 28, 2004 at 08:19:15, Stuart Cracraft wrote: >>>>>>>>>>> >>>>>>>>>>>>On September 28, 2004 at 02:14:51, martin fierz wrote: >>>>>>>>>>>> >>>>>>>>>>>>>On September 27, 2004 at 23:45:54, Stuart Cracraft wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>>I experimented with reordering root ply at iterative depth iply > 1 >>>>>>>>>>>>>>where 1 is the root ply, with the results of iply-1 sorted by the >>>>>>>>>>>>>>total nodes of quiescence and main search defined as the # of entries >>>>>>>>>>>>>>for each of those subroutines. >>>>>>>>>>>>>> >>>>>>>>>>>>>>I didn't sort at root node on the first sort by quiescence but instead >>>>>>>>>>>>>>by my normal scheme though I tried quiescence and it was worse. I felt >>>>>>>>>>>>>>this gave a better chance to the above method. >>>>>>>>>>>>>> >>>>>>>>>>>>>>I sorted moves at the root ply for iply > 1 in the following way >>>>>>>>>>>>>>for 7 different parts to the experiment. >>>>>>>>>>>>>> >>>>>>>>>>>>>> sort by normal method (history heuristic, mvv/lva, see, etc. >>>>>>>>>>>>>> sort exactly by subtree node count, nothing else >>>>>>>>>>>>>> sort by subtree node count added to normal score (hh, mvv/lva, see, etc.) >>>>>>>>>>>>>> same as previous but node count x 10 before addition >>>>>>>>>>>>>> same as previous but node count x 100 before addition >>>>>>>>>>>>>> same as previous but node count x 1000 before addition >>>>>>>>>>>>>> same as previous but node count x 10000 before addition >>>>>>>>>>>>>> >>>>>>>>>>>>>>The results, measured by # right on Win-at-Chess varied from >>>>>>>>>>>>>>250 for the first in the list above to 234 for the last. >>>>>>>>>>>>>>Most bunched up between 244-247 except the first was 250, >>>>>>>>>>>>>>my current best on WAC with handtuning everything. >>>>>>>>>>>>>> >>>>>>>>>>>>>>For me, I'm convinced that this style of sorting root ply is >>>>>>>>>>>>>>slightly less good for my short searches compared to what I am using: >>>>>>>>>>>>>>a combination of history, heuristic, see(), and centrality with >>>>>>>>>>>>>>various bonuses, about a half page of code sprinkled about. >>>>>>>>>>>>>> >>>>>>>>>>>>>>The advantage of sorting root node by subtree is the simplicity. >>>>>>>>>>>>>>It eliminates about a half a page of code and introduces >>>>>>>>>>>>>>about a quarter page of code for only slightly lesser results >>>>>>>>>>>>>>(within 1-2% of my current result) so that is good. >>>>>>>>>>>>>> >>>>>>>>>>>>>>Still I think I'll leave it #ifdefed out for now and use it as >>>>>>>>>>>>>>a baseline that is only improvable upon with handtuning of my >>>>>>>>>>>>>>current methods and others to be discovered. >>>>>>>>>>>>>> >>>>>>>>>>>>>>Stuart >>>>>>>>>>>>> >>>>>>>>>>>>>...as ed schröder said to me: "terrible testing". he was right, of course. >>>>>>>>>>>>> >>>>>>>>>>>>>cheers >>>>>>>>>>>>> martin >>>>>>>>>>>> >>>>>>>>>>>>Each to his own. >>>>>>>>>>> >>>>>>>>>>>if you get free advice from one of the world's best computer chess programmers >>>>>>>>>>>it is a good idea to use it. there's not much point writing tons of posts here >>>>>>>>>>>asking for advice if you don't listen.... >>>>>>>>>>> >>>>>>>>>>>cheers >>>>>>>>>>> martin >>>>>>>>>> >>>>>>>>>>Well, condemnations aside, without specific feedback beyond "Oh that's just >>>>>>>>>>bad" (I can get that at work from the boss or from relatives) -- I don't >>>>>>>>>>respond well to that kind of input. It is non-constructive. >>>>>>>>> >>>>>>>>>my post was meant very constructively :-) >>>>>>>>>i just posted something about root move ordering a day or two ago, and ed >>>>>>>>>schröder answered "terrible testing" with a short explanation of why. i expected >>>>>>>>>you had read that thread, and knew what i meant. if not, read it now! >>>>>>>>> >>>>>>>>>cheers >>>>>>>>> martin >>>>>>>>> >>>>>>>>>PS: if you are not in the habit of reading posts of some particular persons >>>>>>>>>(like ed, bob etc) on this board, you should get into that too! other people >>>>>>>>>have something to say too of course, but we do have some >>>>>>>>>world-class-chess-programmers here and i try to read everything they write... >>>>>>>> >>>>>>>>Believe me: I read every character, every sentence, every word, every >>>>>>>>comma, every dot of Ed S. and Bob H. >>>>>>> >>>>>>>good - then i hope you also remember what ed had to say about a similar >>>>>>>experiment of mine! >>>>>>> >>>>>>>cheers >>>>>>> martin >>>>>> >>>>>>Refresh me -- I already spent too much time at this forum!!!! >>>>> >>>>>sure: "terrible testing" :-) >>>>> >>>>>seriously: i changed my root move ordering and reported results in a) 2x40 games >>>>>against other engines and b) in ECMGCP testsuite. ed made the statement i quoted >>>>>above. >>>>> >>>>>he says: if you only change move ordering, you should keep a testset handy which >>>>>you search to fixed depth, and where you can compare how move ordering affects >>>>>the node count on your test set. best not to compare total nodes, because one >>>>>bad position can skew your results, take e.g. #of positions with less nodes vs >>>>>#of positions with more nodes; or take the geometric mean over the test set of >>>>>newnodes/oldnodes. >>>>> >>>>>this is much a better test of move ordering, because it does not rely on your >>>>>engine accidentally finding or not finding a certain move. >>>>> >>>>>that is what ed says, and i believe he is right :-) >>>>> >>>>>hmm, i could have written this to start with and spared us a lot of >>>>>ping-pong-posts ;-) >>>>> >>>>>cheers >>>>> martin >>>> >>>>Yes -- I remember Ed's comments -- I plan to implement it. Another >>>>one is Bob's idea to complete 1 ply searches but let the current ply >>>>finish before returning. >>>> >>>>Good stuff. >>>> >>>>Stuart >>> >>> >>>That will help a lot, but it won't cure the problem as Martin pointed out. You >>>still have enough timing variance to barely start the next iteration one time >>>but not the next, which will make the results incomparable... >>> >>>But if you are making decisions that affect the _size_ of the tree (ie different >>>root move ordering, or search extensions) then you have to actually _measure_ >>>the size of the tree, not something else. Using a timed search is measuring >>>something else entirely, not the thing directly influenced by your changes... >> >>Absolutely -- but I think either that or bumping up from 1 second per >>move to a few seconds per move, or simply buying the fastest machine >>out there now would do take care of the concerns. > > >You must have a reproducible methodology for testing basic changes, or else this >is a hopeless endeavor. The only way for reproducibility is to search to a >fixed depth, when you are making changes whose sole purpose is to shrink the >tree and therefore the search time... That is what you are missing. >"reproducibility". And without it, improvements are going to be _very_ >difficult to come by and validate. Tree size is not the only measure . The reason I say that is that my recent change to not do check-evasions extensions but instead to do check-giving extensions *increased* my search tree for WAC from 60M nodes to 80M nodes but at the same time nicely increased my result. Had I eliminated changes based on tree size, I wouldn't have gotten this new-sytle program. It has to be a variety of measures. Stuart
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.