Author: Robert Hyatt
Date: 12:31:17 10/02/04
Go up one level in this thread
On October 02, 2004 at 10:29:33, Stuart Cracraft wrote: >On October 01, 2004 at 21:51:55, Robert Hyatt wrote: > >>On October 01, 2004 at 19:40:18, Stuart Cracraft wrote: >> >>>On October 01, 2004 at 00:04:03, Robert Hyatt wrote: >>> >>>>On September 29, 2004 at 19:50:15, Stuart Cracraft wrote: >>>> >>>>>On September 29, 2004 at 17:47:27, martin fierz wrote: >>>>> >>>>>>On September 29, 2004 at 13:44:50, Stuart Cracraft wrote: >>>>>> >>>>>>>On September 29, 2004 at 08:22:38, martin fierz wrote: >>>>>>> >>>>>>>>On September 28, 2004 at 23:49:01, Stuart Cracraft wrote: >>>>>>>> >>>>>>>>>On September 28, 2004 at 16:29:29, martin fierz wrote: >>>>>>>>> >>>>>>>>>>On September 28, 2004 at 13:43:48, Stuart Cracraft wrote: >>>>>>>>>> >>>>>>>>>>>On September 28, 2004 at 08:44:04, martin fierz wrote: >>>>>>>>>>> >>>>>>>>>>>>On September 28, 2004 at 08:19:15, Stuart Cracraft wrote: >>>>>>>>>>>> >>>>>>>>>>>>>On September 28, 2004 at 02:14:51, martin fierz wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>>On September 27, 2004 at 23:45:54, Stuart Cracraft wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>>I experimented with reordering root ply at iterative depth iply > 1 >>>>>>>>>>>>>>>where 1 is the root ply, with the results of iply-1 sorted by the >>>>>>>>>>>>>>>total nodes of quiescence and main search defined as the # of entries >>>>>>>>>>>>>>>for each of those subroutines. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>I didn't sort at root node on the first sort by quiescence but instead >>>>>>>>>>>>>>>by my normal scheme though I tried quiescence and it was worse. I felt >>>>>>>>>>>>>>>this gave a better chance to the above method. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>I sorted moves at the root ply for iply > 1 in the following way >>>>>>>>>>>>>>>for 7 different parts to the experiment. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> sort by normal method (history heuristic, mvv/lva, see, etc. >>>>>>>>>>>>>>> sort exactly by subtree node count, nothing else >>>>>>>>>>>>>>> sort by subtree node count added to normal score (hh, mvv/lva, see, etc.) >>>>>>>>>>>>>>> same as previous but node count x 10 before addition >>>>>>>>>>>>>>> same as previous but node count x 100 before addition >>>>>>>>>>>>>>> same as previous but node count x 1000 before addition >>>>>>>>>>>>>>> same as previous but node count x 10000 before addition >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>The results, measured by # right on Win-at-Chess varied from >>>>>>>>>>>>>>>250 for the first in the list above to 234 for the last. >>>>>>>>>>>>>>>Most bunched up between 244-247 except the first was 250, >>>>>>>>>>>>>>>my current best on WAC with handtuning everything. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>For me, I'm convinced that this style of sorting root ply is >>>>>>>>>>>>>>>slightly less good for my short searches compared to what I am using: >>>>>>>>>>>>>>>a combination of history, heuristic, see(), and centrality with >>>>>>>>>>>>>>>various bonuses, about a half page of code sprinkled about. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>The advantage of sorting root node by subtree is the simplicity. >>>>>>>>>>>>>>>It eliminates about a half a page of code and introduces >>>>>>>>>>>>>>>about a quarter page of code for only slightly lesser results >>>>>>>>>>>>>>>(within 1-2% of my current result) so that is good. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Still I think I'll leave it #ifdefed out for now and use it as >>>>>>>>>>>>>>>a baseline that is only improvable upon with handtuning of my >>>>>>>>>>>>>>>current methods and others to be discovered. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Stuart >>>>>>>>>>>>>> >>>>>>>>>>>>>>...as ed schröder said to me: "terrible testing". he was right, of course. >>>>>>>>>>>>>> >>>>>>>>>>>>>>cheers >>>>>>>>>>>>>> martin >>>>>>>>>>>>> >>>>>>>>>>>>>Each to his own. >>>>>>>>>>>> >>>>>>>>>>>>if you get free advice from one of the world's best computer chess programmers >>>>>>>>>>>>it is a good idea to use it. there's not much point writing tons of posts here >>>>>>>>>>>>asking for advice if you don't listen.... >>>>>>>>>>>> >>>>>>>>>>>>cheers >>>>>>>>>>>> martin >>>>>>>>>>> >>>>>>>>>>>Well, condemnations aside, without specific feedback beyond "Oh that's just >>>>>>>>>>>bad" (I can get that at work from the boss or from relatives) -- I don't >>>>>>>>>>>respond well to that kind of input. It is non-constructive. >>>>>>>>>> >>>>>>>>>>my post was meant very constructively :-) >>>>>>>>>>i just posted something about root move ordering a day or two ago, and ed >>>>>>>>>>schröder answered "terrible testing" with a short explanation of why. i expected >>>>>>>>>>you had read that thread, and knew what i meant. if not, read it now! >>>>>>>>>> >>>>>>>>>>cheers >>>>>>>>>> martin >>>>>>>>>> >>>>>>>>>>PS: if you are not in the habit of reading posts of some particular persons >>>>>>>>>>(like ed, bob etc) on this board, you should get into that too! other people >>>>>>>>>>have something to say too of course, but we do have some >>>>>>>>>>world-class-chess-programmers here and i try to read everything they write... >>>>>>>>> >>>>>>>>>Believe me: I read every character, every sentence, every word, every >>>>>>>>>comma, every dot of Ed S. and Bob H. >>>>>>>> >>>>>>>>good - then i hope you also remember what ed had to say about a similar >>>>>>>>experiment of mine! >>>>>>>> >>>>>>>>cheers >>>>>>>> martin >>>>>>> >>>>>>>Refresh me -- I already spent too much time at this forum!!!! >>>>>> >>>>>>sure: "terrible testing" :-) >>>>>> >>>>>>seriously: i changed my root move ordering and reported results in a) 2x40 games >>>>>>against other engines and b) in ECMGCP testsuite. ed made the statement i quoted >>>>>>above. >>>>>> >>>>>>he says: if you only change move ordering, you should keep a testset handy which >>>>>>you search to fixed depth, and where you can compare how move ordering affects >>>>>>the node count on your test set. best not to compare total nodes, because one >>>>>>bad position can skew your results, take e.g. #of positions with less nodes vs >>>>>>#of positions with more nodes; or take the geometric mean over the test set of >>>>>>newnodes/oldnodes. >>>>>> >>>>>>this is much a better test of move ordering, because it does not rely on your >>>>>>engine accidentally finding or not finding a certain move. >>>>>> >>>>>>that is what ed says, and i believe he is right :-) >>>>>> >>>>>>hmm, i could have written this to start with and spared us a lot of >>>>>>ping-pong-posts ;-) >>>>>> >>>>>>cheers >>>>>> martin >>>>> >>>>>Yes -- I remember Ed's comments -- I plan to implement it. Another >>>>>one is Bob's idea to complete 1 ply searches but let the current ply >>>>>finish before returning. >>>>> >>>>>Good stuff. >>>>> >>>>>Stuart >>>> >>>> >>>>That will help a lot, but it won't cure the problem as Martin pointed out. You >>>>still have enough timing variance to barely start the next iteration one time >>>>but not the next, which will make the results incomparable... >>>> >>>>But if you are making decisions that affect the _size_ of the tree (ie different >>>>root move ordering, or search extensions) then you have to actually _measure_ >>>>the size of the tree, not something else. Using a timed search is measuring >>>>something else entirely, not the thing directly influenced by your changes... >>> >>>Absolutely -- but I think either that or bumping up from 1 second per >>>move to a few seconds per move, or simply buying the fastest machine >>>out there now would do take care of the concerns. >> >> >>You must have a reproducible methodology for testing basic changes, or else this >>is a hopeless endeavor. The only way for reproducibility is to search to a >>fixed depth, when you are making changes whose sole purpose is to shrink the >>tree and therefore the search time... That is what you are missing. >>"reproducibility". And without it, improvements are going to be _very_ >>difficult to come by and validate. > >Tree size is not the only measure . The reason I say that is that my >recent change to not do check-evasions extensions but instead to do >check-giving extensions *increased* my search tree for WAC from 60M >nodes to 80M nodes but at the same time nicely increased my result. Something is _terribly_ wrong with your testing. if you search for 1 sec/move, and suddenly search 33% _more_ nodes, that must mean that suddenly your search speed (NPS) is _also_ 33% faster? No way to search to fixed 1 sec/position, increase the total nodes searched by 33%, without getting a lot faster in your search. As I said, _something_ doesn't jive... > >Had I eliminated changes based on tree size, I wouldn't have gotten >this new-sytle program. It has to be a variety of measures. > >Stuart
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.