Author: Stuart Cracraft
Date: 10:53:53 10/04/04
Go up one level in this thread
On October 02, 2004 at 23:22:53, Robert Hyatt wrote: >On October 02, 2004 at 17:04:36, Stuart Cracraft wrote: > >>On October 02, 2004 at 15:31:17, Robert Hyatt wrote: >> >>>On October 02, 2004 at 10:29:33, Stuart Cracraft wrote: >>> >>>>On October 01, 2004 at 21:51:55, Robert Hyatt wrote: >>>> >>>>>On October 01, 2004 at 19:40:18, Stuart Cracraft wrote: >>>>> >>>>>>On October 01, 2004 at 00:04:03, Robert Hyatt wrote: >>>>>> >>>>>>>On September 29, 2004 at 19:50:15, Stuart Cracraft wrote: >>>>>>> >>>>>>>>On September 29, 2004 at 17:47:27, martin fierz wrote: >>>>>>>> >>>>>>>>>On September 29, 2004 at 13:44:50, Stuart Cracraft wrote: >>>>>>>>> >>>>>>>>>>On September 29, 2004 at 08:22:38, martin fierz wrote: >>>>>>>>>> >>>>>>>>>>>On September 28, 2004 at 23:49:01, Stuart Cracraft wrote: >>>>>>>>>>> >>>>>>>>>>>>On September 28, 2004 at 16:29:29, martin fierz wrote: >>>>>>>>>>>> >>>>>>>>>>>>>On September 28, 2004 at 13:43:48, Stuart Cracraft wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>>On September 28, 2004 at 08:44:04, martin fierz wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>>On September 28, 2004 at 08:19:15, Stuart Cracraft wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>On September 28, 2004 at 02:14:51, martin fierz wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>On September 27, 2004 at 23:45:54, Stuart Cracraft wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>I experimented with reordering root ply at iterative depth iply > 1 >>>>>>>>>>>>>>>>>>where 1 is the root ply, with the results of iply-1 sorted by the >>>>>>>>>>>>>>>>>>total nodes of quiescence and main search defined as the # of entries >>>>>>>>>>>>>>>>>>for each of those subroutines. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>I didn't sort at root node on the first sort by quiescence but instead >>>>>>>>>>>>>>>>>>by my normal scheme though I tried quiescence and it was worse. I felt >>>>>>>>>>>>>>>>>>this gave a better chance to the above method. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>I sorted moves at the root ply for iply > 1 in the following way >>>>>>>>>>>>>>>>>>for 7 different parts to the experiment. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> sort by normal method (history heuristic, mvv/lva, see, etc. >>>>>>>>>>>>>>>>>> sort exactly by subtree node count, nothing else >>>>>>>>>>>>>>>>>> sort by subtree node count added to normal score (hh, mvv/lva, see, etc.) >>>>>>>>>>>>>>>>>> same as previous but node count x 10 before addition >>>>>>>>>>>>>>>>>> same as previous but node count x 100 before addition >>>>>>>>>>>>>>>>>> same as previous but node count x 1000 before addition >>>>>>>>>>>>>>>>>> same as previous but node count x 10000 before addition >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>The results, measured by # right on Win-at-Chess varied from >>>>>>>>>>>>>>>>>>250 for the first in the list above to 234 for the last. >>>>>>>>>>>>>>>>>>Most bunched up between 244-247 except the first was 250, >>>>>>>>>>>>>>>>>>my current best on WAC with handtuning everything. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>For me, I'm convinced that this style of sorting root ply is >>>>>>>>>>>>>>>>>>slightly less good for my short searches compared to what I am using: >>>>>>>>>>>>>>>>>>a combination of history, heuristic, see(), and centrality with >>>>>>>>>>>>>>>>>>various bonuses, about a half page of code sprinkled about. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>The advantage of sorting root node by subtree is the simplicity. >>>>>>>>>>>>>>>>>>It eliminates about a half a page of code and introduces >>>>>>>>>>>>>>>>>>about a quarter page of code for only slightly lesser results >>>>>>>>>>>>>>>>>>(within 1-2% of my current result) so that is good. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Still I think I'll leave it #ifdefed out for now and use it as >>>>>>>>>>>>>>>>>>a baseline that is only improvable upon with handtuning of my >>>>>>>>>>>>>>>>>>current methods and others to be discovered. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Stuart >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>...as ed schröder said to me: "terrible testing". he was right, of course. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>cheers >>>>>>>>>>>>>>>>> martin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Each to his own. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>if you get free advice from one of the world's best computer chess programmers >>>>>>>>>>>>>>>it is a good idea to use it. there's not much point writing tons of posts here >>>>>>>>>>>>>>>asking for advice if you don't listen.... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>cheers >>>>>>>>>>>>>>> martin >>>>>>>>>>>>>> >>>>>>>>>>>>>>Well, condemnations aside, without specific feedback beyond "Oh that's just >>>>>>>>>>>>>>bad" (I can get that at work from the boss or from relatives) -- I don't >>>>>>>>>>>>>>respond well to that kind of input. It is non-constructive. >>>>>>>>>>>>> >>>>>>>>>>>>>my post was meant very constructively :-) >>>>>>>>>>>>>i just posted something about root move ordering a day or two ago, and ed >>>>>>>>>>>>>schröder answered "terrible testing" with a short explanation of why. i expected >>>>>>>>>>>>>you had read that thread, and knew what i meant. if not, read it now! >>>>>>>>>>>>> >>>>>>>>>>>>>cheers >>>>>>>>>>>>> martin >>>>>>>>>>>>> >>>>>>>>>>>>>PS: if you are not in the habit of reading posts of some particular persons >>>>>>>>>>>>>(like ed, bob etc) on this board, you should get into that too! other people >>>>>>>>>>>>>have something to say too of course, but we do have some >>>>>>>>>>>>>world-class-chess-programmers here and i try to read everything they write... >>>>>>>>>>>> >>>>>>>>>>>>Believe me: I read every character, every sentence, every word, every >>>>>>>>>>>>comma, every dot of Ed S. and Bob H. >>>>>>>>>>> >>>>>>>>>>>good - then i hope you also remember what ed had to say about a similar >>>>>>>>>>>experiment of mine! >>>>>>>>>>> >>>>>>>>>>>cheers >>>>>>>>>>> martin >>>>>>>>>> >>>>>>>>>>Refresh me -- I already spent too much time at this forum!!!! >>>>>>>>> >>>>>>>>>sure: "terrible testing" :-) >>>>>>>>> >>>>>>>>>seriously: i changed my root move ordering and reported results in a) 2x40 games >>>>>>>>>against other engines and b) in ECMGCP testsuite. ed made the statement i quoted >>>>>>>>>above. >>>>>>>>> >>>>>>>>>he says: if you only change move ordering, you should keep a testset handy which >>>>>>>>>you search to fixed depth, and where you can compare how move ordering affects >>>>>>>>>the node count on your test set. best not to compare total nodes, because one >>>>>>>>>bad position can skew your results, take e.g. #of positions with less nodes vs >>>>>>>>>#of positions with more nodes; or take the geometric mean over the test set of >>>>>>>>>newnodes/oldnodes. >>>>>>>>> >>>>>>>>>this is much a better test of move ordering, because it does not rely on your >>>>>>>>>engine accidentally finding or not finding a certain move. >>>>>>>>> >>>>>>>>>that is what ed says, and i believe he is right :-) >>>>>>>>> >>>>>>>>>hmm, i could have written this to start with and spared us a lot of >>>>>>>>>ping-pong-posts ;-) >>>>>>>>> >>>>>>>>>cheers >>>>>>>>> martin >>>>>>>> >>>>>>>>Yes -- I remember Ed's comments -- I plan to implement it. Another >>>>>>>>one is Bob's idea to complete 1 ply searches but let the current ply >>>>>>>>finish before returning. >>>>>>>> >>>>>>>>Good stuff. >>>>>>>> >>>>>>>>Stuart >>>>>>> >>>>>>> >>>>>>>That will help a lot, but it won't cure the problem as Martin pointed out. You >>>>>>>still have enough timing variance to barely start the next iteration one time >>>>>>>but not the next, which will make the results incomparable... >>>>>>> >>>>>>>But if you are making decisions that affect the _size_ of the tree (ie different >>>>>>>root move ordering, or search extensions) then you have to actually _measure_ >>>>>>>the size of the tree, not something else. Using a timed search is measuring >>>>>>>something else entirely, not the thing directly influenced by your changes... >>>>>> >>>>>>Absolutely -- but I think either that or bumping up from 1 second per >>>>>>move to a few seconds per move, or simply buying the fastest machine >>>>>>out there now would do take care of the concerns. >>>>> >>>>> >>>>>You must have a reproducible methodology for testing basic changes, or else this >>>>>is a hopeless endeavor. The only way for reproducibility is to search to a >>>>>fixed depth, when you are making changes whose sole purpose is to shrink the >>>>>tree and therefore the search time... That is what you are missing. >>>>>"reproducibility". And without it, improvements are going to be _very_ >>>>>difficult to come by and validate. >>>> >>>>Tree size is not the only measure . The reason I say that is that my >>>>recent change to not do check-evasions extensions but instead to do >>>>check-giving extensions *increased* my search tree for WAC from 60M >>>>nodes to 80M nodes but at the same time nicely increased my result. >>> >>> >>>Something is _terribly_ wrong with your testing. if you search for 1 sec/move, >>>and suddenly search 33% _more_ nodes, that must mean that suddenly your search >>>speed (NPS) is _also_ 33% faster? >>> >>>No way to search to fixed 1 sec/position, increase the total nodes searched by >>>33%, without getting a lot faster in your search. >>> >>>As I said, _something_ doesn't jive... >>> >>> >>>> >>>>Had I eliminated changes based on tree size, I wouldn't have gotten >>>>this new-sytle program. It has to be a variety of measures. >>>> >>>>Stuart >> >>I made a change to finish the current iteration rather than aborting >>at 1 second. >> >>That change made the total 300 seconds for WAC @ 1 second each >>take almost double the time to between 500-600. >> >>I don't know about NPS but that's what happened. > >Then what is this: > > >>>>Tree size is not the only measure . The reason I say that is that my >>>>recent change to not do check-evasions extensions but instead to do >>>>check-giving extensions *increased* my search tree for WAC from 60M >>>>nodes to 80M nodes but at the same time nicely increased my result. > >IE that suggests your tree size was increased by the check evasion changes, but >the previous statement suggests the tree size was increased by changing your >search to complete the current iteration... Sorry my mistake. I was comparing apples with oranges. I reran the test and the node count is substantially less for extending all moves that check in main search over extending all moves that escape check in main search. 10% less. Plus the result is substantially better. This was on the first 30 problems in WAC. Extend all moves that check in main search, only captures in quiescence + 8.47/17.83 96% 29/30 ha=42 30.54 8874134 295804/1/290612 0/57583/0/202082/4153 078/0 Extend all moves that escape check in main search, only captures in quiescence + 8.97/17.07 90% 27/30 ha=41 30.26 9838794 327960/1/325163 0/55737/0/96236/45657 00/0 What I had done earlier was to not separate out the tests so as to compare just those two, devoid of other changes.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.