Author: Daniel Shawul
Date: 22:39:56 06/17/05
Go up one level in this thread
On June 17, 2005 at 19:10:08, Vincent Diepeveen wrote: >On June 17, 2005 at 01:52:45, Daniel Shawul wrote: > >>On June 16, 2005 at 18:01:58, Vincent Diepeveen wrote: >> >>>On June 15, 2005 at 16:36:54, Eric Oldre wrote: >>> >>>Of course the more time you invest in debugging the better. >>> >> >>Hi >>i have tried to output the tree to disk,when i tried parallel >>tree searching using PV split method. Thread1 outputs to file1,and >>Thread2 to file2. But the output produced is unreadable. >>you might have already explanined how to do this, but i couldn't get it. >>best >>daniel > >For parallel search what really debugged all my bugs is next : > >at startup i allocate 1 big shared memory block which is a huge array of >integers. additional i have 1 global volatile lock in the shared memory. > >each processor is doing next in search. If they hit at a line that writes debug >info, then i'm doing basically this : > >In the code you will find for example this : > > #if DEBUGNODESVERBOSE > dverb_info[0] = (int)rb->idnumber; > dverb_info[1] = rb->alfa; > dverb_info[2] = rb->beta; > dverb_info[3] = rb->ply; // depthleft > DVERB(4,dverb_abigetblock); > #endif > >Now the function DVERB is doing this : >void DVERB(int ninfo,int dverbflag) { > int header,i; > /* header byte: > * 32 bits waarvan: > * 1 sign (unused) > * 17 laatste debugsmp punt (indien aanwezig) > * 10 verbosepoint: dverb_... > * 4 processornummer (max 0..15) > */ > //printf("locking dverb ninfo=%i flag=%i\n",ninfo,dverbflag); > Lock(tree->lock_dva); > header = 0; > //#endif > header |= (dverbflag<<4)|ProcessNumber; > dva[tree->dva_count++] = header; > i = 0; > while( i < ninfo ) { > dva[tree->dva_count++] = dverb_info[i]; > i++; > } > > if( tree->dva_count >= DVASIZE-64 ) { > int tdr; > > printf("het probleem is duidelijk!\n"); > tdr = tree->dva_rond+1; > // printf("Passed DVASIZE (%i bytes) already %i >times!\n",sizeof(int)*tree->dva_count,tdr); > if( nprocesses == 1 && false ) { > //printf("gonna save it now\n"); > Unlock(tree->lock_dva); > SMPdebugSAVE(); > InitVerbose(); > Lock(tree->lock_dva); > } > else { > int m,move; > //mprint("processes = %i rounds = %i\n",tree->nprocesses,tdr); > InitVerbose(); > /* nu vastleggen het volgende: de zetstack van de verschillende processors >*/ > dva[tree->dva_count++] = (dverb_movestacks<<4); > for( i = 0; i < nprocesses; i++ ) { > dva[tree->dva_count++] = 5; > m = 0; > //mprint("stored line in DVA: %i ... ",tree->movesmadeindex[i]); > while( m < 5 ) { > move = m; > mprint("%s%s ",M2A[(move>>6)&63],M2A[move&63]); > dva[tree->dva_count++] = move; > m++; > } > //mprint("\n"); > } > } > tree->dva_rond = tdr; > } > Unlock(tree->lock_dva); >} > >So basically lock, store the search info as integers in the array and unlock. >When it has crashed, all you have to do is write the shared memory to disk and >then parse the shared memory until you found the processor that crashed and you >figure out where it was and what it was doing and the state at which other >processors were at that moment and so on. > >And then pray that you crash very quickly. Basically when this testbed works >fine all you have got to do is find the quickest way to crash or get a deadlock >or race condition in your program. Then debug backwards. > >That's not so easy and requires a lot of experience :) > >But you can get very complex parallel forms of search to work with this method. >Because basically every processor directly locks and unlocks, the odds are not >so big that this 'debug code' in shared memory will eat all system time. > >Your evaluation will, your move ordering will and so on. > >It's just a sequential fill of shared memory. > >Then additional write tools to find relevant data within that huge file. > >Ways to really quickly crash your program is important to find too. A good one >i'll give for free, that's use the utmost stupid way to split, like splitting >always and especially as much near the leaves as possible. And start searching >with 16 cpu's :) > >That really crashes the program quickly :) > >Now the way in which i print out things. To give a real example of 16 processors >(note my printing routines aren't even good working for it) : > >0............... SplitInOthers: processnumber=0 newproc=8 idnumberposition=1236 >0............... SplitInOthers: processnumber=0 newproc=15 idnumberposition=123 >6 >0............... SplitInOthers: processnumber=0 newproc=1 idnumberposition=1236 >..........10..... Splitting position id=1236 >..........10..... Splitting position: found and moved to position 1236 >.......7........ Splitting position id=1236 >.........9...... Splitting position id=1236 >.........9...... Splitting position: found and moved to position 1236 >.......7........ Splitting position: found and moved to position 1236 >.....5.......... Splitting position id=1236 >.....5.......... Splitting position: found and moved to position 1236 >..2............. Splitting position id=1236 >..2............. Splitting position: found and moved to position 1236 >...........11.... Splitting position id=1236 >...........11.... Splitting position: found and moved to position 1236 >........8....... Splitting position id=1236 >........8....... Splitting position: found and moved to position 1236 >............12... Splitting position id=1236 >............12... Splitting position: found and moved to position 1236 >...3............ Splitting position id=1236 >...3............ Splitting position: found and moved to position 1236 >......6......... Splitting position id=1236 >......6......... Splitting position: found and moved to position 1236 >....4........... Splitting position id=1236 >.............13.. Splitting position id=1236 >..............14. Splitting position id=1236 >....4........... Splitting position: found and moved to position 1236 >...............15 Splitting position id=1236 >.1.............. Splitting position id=1236 >..............14. Splitting position: found and moved to position 1236 >...............15 Splitting position: found and moved to position 1236 >.............13.. Splitting position: found and moved to position 1236 >.1.............. Splitting position: found and moved to position 1236 >............12... Abortme() id=1236 >............12... splitid=0 depth=5 procs=[0,6,10,3,2,-1,9,13,4,11,5,7,14,8,15, >1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >............12... PartBoth: proc just returned to idle searching 0 nodes >............12... ActionCheck() flags=1 realply=0 >.....5.......... Abortme() id=1236 >.....5.......... splitid=0 depth=5 procs=[0,6,10,3,2,-1,9,13,4,11,-1,7,14,8,15, >1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >.....5.......... PartBoth: proc just returned to idle searching 0 nodes >.....5.......... ActionCheck() flags=1 realply=0 >..............14. Abortme() id=1236 >..............14. splitid=0 depth=5 procs=[0,6,10,3,2,-1,9,13,4,11,-1,7,-1,8,15 >,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >..............14. PartBoth: proc just returned to idle searching 0 nodes >..............14. ActionCheck() flags=1 realply=0 >...3............ Abortme() id=1236 >...3............ splitid=0 depth=5 procs=[0,6,10,-1,2,-1,9,13,4,11,-1,7,-1,8,15 >,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >...3............ PartBoth: proc just returned to idle searching 0 nodes >...3............ ActionCheck() flags=1 realply=0 >......6......... Abortme() id=1236 >......6......... splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,13,4,11,-1,7,-1,8,1 >5,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >......6......... PartBoth: proc just returned to idle searching 0 nodes >......6......... ActionCheck() flags=1 realply=0 >.............13.. Abortme() id=1236 >.............13.. splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,4,11,-1,7,-1,8, >15,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >.............13.. PartBoth: proc just returned to idle searching 0 nodes >.............13.. ActionCheck() flags=1 realply=0 >....4........... Abortme() id=1236 >....4........... splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8, >15,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >....4........... PartBoth: proc just returned to idle searching 0 nodes >....4........... ActionCheck() flags=1 realply=0 >.1.............. Abortme() id=1236 >.1.............. splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8, >15,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >.1.............. PartBoth: proc just returned to idle searching 0 nodes >.1.............. ActionCheck() flags=1 realply=0 >...............15 Abortme() id=1236 >...............15 splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8 >,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >...............15 PartBoth: proc just returned to idle searching 0 nodes >...............15 ActionCheck() flags=1 realply=0 >0............... Abortme() id=1236 >0............... splitid=0 depth=5 procs=[-1,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8 >,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >0............... PartBoth: proc just returned to idle searching 0 nodes >0............... ActionCheck() flags=1 realply=0 >..........10..... Abortme() id=1236 >..........10..... splitid=0 depth=5 procs=[-1,-1,-1,-1,2,-1,9,-1,-1,11,-1,7,-1, >8,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >..........10..... PartBoth: proc just returned to idle searching 0 nodes >..........10..... ActionCheck() flags=1 realply=0 >..2............. Abortme() id=1236 >..2............. splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,11,-1,7,-1, >8,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >..2............. ActionCheck() flags=1 realply=0 >.......7........ Abortme() id=1236 >.......7........ splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,11,-1,-1,-1 >,8,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >.......7........ PartBoth: proc just returned to idle searching 0 nodes >.......7........ ActionCheck() flags=1 realply=0 >........8....... Abortme() id=1236 >........8....... splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,11,-1,-1,-1 >,-1,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >........8....... PartBoth: proc just returned to idle searching 0 nodes >........8....... ActionCheck() flags=1 realply=0 >...........11.... Abortme() id=1236 >...........11.... splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,-1,-1,-1,- >1,-1,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >...........11.... PartBoth: proc just returned to idle searching 0 nodes >...........11.... ActionCheck() flags=1 realply=0 >.........9...... MakeSplittable() id=1233 realply=3 >.........9...... registeridlesplits owner proc=9 >.........9...... splitid=0 depth=3 procs=[11,8,7,2,10,0,9,15,1,4,13,6,3,14,5,12 >] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1] >.........9...... SplitInOthers: processnumber=9 newproc=11 idnumberposition=123 >3 >and so on > >As you see this output is just printing all actions about parallel search, not >even printing info regarding the actual searched tree (as that would be hundreds >of megabytes). > >The huge advantage of writing the debug info to shared memory is that it eats so >little time that the processors get more time into their splitting routines and >get therefore bigger odds to crash if there is a bug. > >>>Apart from boundschecking with a real boundschecker; gcc has a great >>>boundschecker extension at their homepage, windows has nothing that impressed me >>>in fact like devpartner doesn't even find the most obvious boundsbug in my GUI. >>>Further there is normal debugging ways, the most effective way in which i debug >>>Diep if there is problems is in the next trivial way: >>> >>>That simply printing the entire search tree to disk. That can be done either >>>direct or indirect : >>> >>>example output (written by hand just to show example now for 4 processors) : >>> >>>0... searching root >>>0... makemove a2a3 >>>0... a2a3 makemove d7d5 >>>0... a2a3 d7d5 eval -0.200 >>>0... a2a3 d7d5 ab=[0.001;0.002] cutoff -0.200 >>>... >>> >>>And when processor 1 is doing an action : >>>.1.. blabla >>> >>>In short this is a very powerful way to debug search or debug parallel bugs. >>>Of course for tough race conditions in a parallel search it is not so clever to >>>directly print to textfile, as the time spent to print is too long (which means >>>a deadlock or race conditions will happen not so soon as the time spent to >>>printing is too huge). >>> >>>So i print it in a shared block of memory (locking first of course) and later >>>parse that memory block to the above text. >>> >>>In this way i managed to debug the parallel search of Diep. >>> >>>For single cpu just printing directly to disk will do fine and you can examine >>>how big your trees get at small search depths up to a ply or 6 is no problem. >>> >>>You'll find many weird stuff in your qsearch and your move ordering like this. >>> >>>Of course it's time consuming to do the above (reading those huge textfiles) but >>>it's very rewarding. >>> >>>The above has another huge advantage and that it is finding bugs in your >>>evaluation function and directly gives you an answer whether a new compiler >>>sucks, is dangerous to use or whether you have a so far unfound bug which >>>boundscheckers didn't find. >>> >>>That is, provided you also print the evaluation score in the textfile. >>> >>>You can compile at different platforms with different type of compilers your >>>code and just compare the textfile verbose. If it is not 100% similar, then >>>obviously there is a bug somewhere. >>> >>>This really will remove about all nasty to find bugs in your entire >>>search&evaluation function. >>> >>>It is the by far the best test to see whether a new compiler has bugs. >>> >>>Especially many intel compilers and certain gcc compilers did not pass this test >>>in past when using -O3 or in case of intel just -O2. gcc 3.3.x and 3.4.x was >>>real real buggy here. 4.0 is far better. >>> >>>Of course the fact that you *seriously* look to debugging methods in your engine >>>already means that your thing is going to be more bugfree than majority of >>>amateur programs. Most haven't heard in fact from the word 'boundschecker'. >>> >>>The fact that out of any random compiler with this trick i can find >>>optimizations that produce bugs, tells me how poor the different compiler teams >>>are in debugging their compilers. >>> >>>Diep is not using any difficult casting construction or so, i never take risks >>>there. Additional it is 100% integer code, so no floating point tricks are >>>possible for compilers. >>> >>>Especially the GCC team is the most horrible code writing team on planet earth. >>> >>>Like 99% of their releases suffers so much from bugs that i can't even compile >>>with -O3 without catching bugs. >>> >>>Obviously that i can prove that with such a simple debugging method is telling >>>really a lot about how powerful the verbose printing method of a search is. >>> >>>>I was wondering if anyone would like to volunteer any tricks they've >>>>used to help find certain bugs in their search function. >>> >>>>I think that the ideas of using perft for move generation and >>>>reversing the board to find bugs in the evaluation have both >>>>been really useful to me. I was wondering if anyone has used >>>>techniques similar to these to help find search bugs. >>>> >>>>I understand that just because a engine can properly pass these >>>>and other tests doesn't mean it's free of bugs. But they certainly >>>>help detect at least some of them. >>>> >>>>I'm certain that there must be plenty of bugs in Latista's search >>>>and I think it's time for me to work on discovering them. If >>>>you don't have any automated tricks like above. Does anyone >>>>have any general advise to help me spot some bugs? >>>> >>>>Eric Oldre >>>> >>>>PS. I have at various spots in my program tried to follow a similar >>>>model of asserts as in fruit. I'm sure taking some time to >>>>do this at more parts of my program would help.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.