Author: Stuart Cracraft
Date: 14:04:36 10/02/04
Go up one level in this thread
On October 02, 2004 at 15:31:17, Robert Hyatt wrote: >On October 02, 2004 at 10:29:33, Stuart Cracraft wrote: > >>On October 01, 2004 at 21:51:55, Robert Hyatt wrote: >> >>>On October 01, 2004 at 19:40:18, Stuart Cracraft wrote: >>> >>>>On October 01, 2004 at 00:04:03, Robert Hyatt wrote: >>>> >>>>>On September 29, 2004 at 19:50:15, Stuart Cracraft wrote: >>>>> >>>>>>On September 29, 2004 at 17:47:27, martin fierz wrote: >>>>>> >>>>>>>On September 29, 2004 at 13:44:50, Stuart Cracraft wrote: >>>>>>> >>>>>>>>On September 29, 2004 at 08:22:38, martin fierz wrote: >>>>>>>> >>>>>>>>>On September 28, 2004 at 23:49:01, Stuart Cracraft wrote: >>>>>>>>> >>>>>>>>>>On September 28, 2004 at 16:29:29, martin fierz wrote: >>>>>>>>>> >>>>>>>>>>>On September 28, 2004 at 13:43:48, Stuart Cracraft wrote: >>>>>>>>>>> >>>>>>>>>>>>On September 28, 2004 at 08:44:04, martin fierz wrote: >>>>>>>>>>>> >>>>>>>>>>>>>On September 28, 2004 at 08:19:15, Stuart Cracraft wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>>On September 28, 2004 at 02:14:51, martin fierz wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>>On September 27, 2004 at 23:45:54, Stuart Cracraft wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>I experimented with reordering root ply at iterative depth iply > 1 >>>>>>>>>>>>>>>>where 1 is the root ply, with the results of iply-1 sorted by the >>>>>>>>>>>>>>>>total nodes of quiescence and main search defined as the # of entries >>>>>>>>>>>>>>>>for each of those subroutines. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>I didn't sort at root node on the first sort by quiescence but instead >>>>>>>>>>>>>>>>by my normal scheme though I tried quiescence and it was worse. I felt >>>>>>>>>>>>>>>>this gave a better chance to the above method. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>I sorted moves at the root ply for iply > 1 in the following way >>>>>>>>>>>>>>>>for 7 different parts to the experiment. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> sort by normal method (history heuristic, mvv/lva, see, etc. >>>>>>>>>>>>>>>> sort exactly by subtree node count, nothing else >>>>>>>>>>>>>>>> sort by subtree node count added to normal score (hh, mvv/lva, see, etc.) >>>>>>>>>>>>>>>> same as previous but node count x 10 before addition >>>>>>>>>>>>>>>> same as previous but node count x 100 before addition >>>>>>>>>>>>>>>> same as previous but node count x 1000 before addition >>>>>>>>>>>>>>>> same as previous but node count x 10000 before addition >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>The results, measured by # right on Win-at-Chess varied from >>>>>>>>>>>>>>>>250 for the first in the list above to 234 for the last. >>>>>>>>>>>>>>>>Most bunched up between 244-247 except the first was 250, >>>>>>>>>>>>>>>>my current best on WAC with handtuning everything. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>For me, I'm convinced that this style of sorting root ply is >>>>>>>>>>>>>>>>slightly less good for my short searches compared to what I am using: >>>>>>>>>>>>>>>>a combination of history, heuristic, see(), and centrality with >>>>>>>>>>>>>>>>various bonuses, about a half page of code sprinkled about. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>The advantage of sorting root node by subtree is the simplicity. >>>>>>>>>>>>>>>>It eliminates about a half a page of code and introduces >>>>>>>>>>>>>>>>about a quarter page of code for only slightly lesser results >>>>>>>>>>>>>>>>(within 1-2% of my current result) so that is good. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Still I think I'll leave it #ifdefed out for now and use it as >>>>>>>>>>>>>>>>a baseline that is only improvable upon with handtuning of my >>>>>>>>>>>>>>>>current methods and others to be discovered. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Stuart >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>...as ed schröder said to me: "terrible testing". he was right, of course. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>cheers >>>>>>>>>>>>>>> martin >>>>>>>>>>>>>> >>>>>>>>>>>>>>Each to his own. >>>>>>>>>>>>> >>>>>>>>>>>>>if you get free advice from one of the world's best computer chess programmers >>>>>>>>>>>>>it is a good idea to use it. there's not much point writing tons of posts here >>>>>>>>>>>>>asking for advice if you don't listen.... >>>>>>>>>>>>> >>>>>>>>>>>>>cheers >>>>>>>>>>>>> martin >>>>>>>>>>>> >>>>>>>>>>>>Well, condemnations aside, without specific feedback beyond "Oh that's just >>>>>>>>>>>>bad" (I can get that at work from the boss or from relatives) -- I don't >>>>>>>>>>>>respond well to that kind of input. It is non-constructive. >>>>>>>>>>> >>>>>>>>>>>my post was meant very constructively :-) >>>>>>>>>>>i just posted something about root move ordering a day or two ago, and ed >>>>>>>>>>>schröder answered "terrible testing" with a short explanation of why. i expected >>>>>>>>>>>you had read that thread, and knew what i meant. if not, read it now! >>>>>>>>>>> >>>>>>>>>>>cheers >>>>>>>>>>> martin >>>>>>>>>>> >>>>>>>>>>>PS: if you are not in the habit of reading posts of some particular persons >>>>>>>>>>>(like ed, bob etc) on this board, you should get into that too! other people >>>>>>>>>>>have something to say too of course, but we do have some >>>>>>>>>>>world-class-chess-programmers here and i try to read everything they write... >>>>>>>>>> >>>>>>>>>>Believe me: I read every character, every sentence, every word, every >>>>>>>>>>comma, every dot of Ed S. and Bob H. >>>>>>>>> >>>>>>>>>good - then i hope you also remember what ed had to say about a similar >>>>>>>>>experiment of mine! >>>>>>>>> >>>>>>>>>cheers >>>>>>>>> martin >>>>>>>> >>>>>>>>Refresh me -- I already spent too much time at this forum!!!! >>>>>>> >>>>>>>sure: "terrible testing" :-) >>>>>>> >>>>>>>seriously: i changed my root move ordering and reported results in a) 2x40 games >>>>>>>against other engines and b) in ECMGCP testsuite. ed made the statement i quoted >>>>>>>above. >>>>>>> >>>>>>>he says: if you only change move ordering, you should keep a testset handy which >>>>>>>you search to fixed depth, and where you can compare how move ordering affects >>>>>>>the node count on your test set. best not to compare total nodes, because one >>>>>>>bad position can skew your results, take e.g. #of positions with less nodes vs >>>>>>>#of positions with more nodes; or take the geometric mean over the test set of >>>>>>>newnodes/oldnodes. >>>>>>> >>>>>>>this is much a better test of move ordering, because it does not rely on your >>>>>>>engine accidentally finding or not finding a certain move. >>>>>>> >>>>>>>that is what ed says, and i believe he is right :-) >>>>>>> >>>>>>>hmm, i could have written this to start with and spared us a lot of >>>>>>>ping-pong-posts ;-) >>>>>>> >>>>>>>cheers >>>>>>> martin >>>>>> >>>>>>Yes -- I remember Ed's comments -- I plan to implement it. Another >>>>>>one is Bob's idea to complete 1 ply searches but let the current ply >>>>>>finish before returning. >>>>>> >>>>>>Good stuff. >>>>>> >>>>>>Stuart >>>>> >>>>> >>>>>That will help a lot, but it won't cure the problem as Martin pointed out. You >>>>>still have enough timing variance to barely start the next iteration one time >>>>>but not the next, which will make the results incomparable... >>>>> >>>>>But if you are making decisions that affect the _size_ of the tree (ie different >>>>>root move ordering, or search extensions) then you have to actually _measure_ >>>>>the size of the tree, not something else. Using a timed search is measuring >>>>>something else entirely, not the thing directly influenced by your changes... >>>> >>>>Absolutely -- but I think either that or bumping up from 1 second per >>>>move to a few seconds per move, or simply buying the fastest machine >>>>out there now would do take care of the concerns. >>> >>> >>>You must have a reproducible methodology for testing basic changes, or else this >>>is a hopeless endeavor. The only way for reproducibility is to search to a >>>fixed depth, when you are making changes whose sole purpose is to shrink the >>>tree and therefore the search time... That is what you are missing. >>>"reproducibility". And without it, improvements are going to be _very_ >>>difficult to come by and validate. >> >>Tree size is not the only measure . The reason I say that is that my >>recent change to not do check-evasions extensions but instead to do >>check-giving extensions *increased* my search tree for WAC from 60M >>nodes to 80M nodes but at the same time nicely increased my result. > > >Something is _terribly_ wrong with your testing. if you search for 1 sec/move, >and suddenly search 33% _more_ nodes, that must mean that suddenly your search >speed (NPS) is _also_ 33% faster? > >No way to search to fixed 1 sec/position, increase the total nodes searched by >33%, without getting a lot faster in your search. > >As I said, _something_ doesn't jive... > > >> >>Had I eliminated changes based on tree size, I wouldn't have gotten >>this new-sytle program. It has to be a variety of measures. >> >>Stuart I made a change to finish the current iteration rather than aborting at 1 second. That change made the total 300 seconds for WAC @ 1 second each take almost double the time to between 500-600. I don't know about NPS but that's what happened.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.