Author: Vincent Diepeveen
Date: 16:10:08 06/17/05
Go up one level in this thread
On June 17, 2005 at 01:52:45, Daniel Shawul wrote:
>On June 16, 2005 at 18:01:58, Vincent Diepeveen wrote:
>
>>On June 15, 2005 at 16:36:54, Eric Oldre wrote:
>>
>>Of course the more time you invest in debugging the better.
>>
>
>Hi
>i have tried to output the tree to disk,when i tried parallel
>tree searching using PV split method. Thread1 outputs to file1,and
>Thread2 to file2. But the output produced is unreadable.
>you might have already explanined how to do this, but i couldn't get it.
>best
>daniel
For parallel search what really debugged all my bugs is next :
at startup i allocate 1 big shared memory block which is a huge array of
integers. additional i have 1 global volatile lock in the shared memory.
each processor is doing next in search. If they hit at a line that writes debug
info, then i'm doing basically this :
In the code you will find for example this :
#if DEBUGNODESVERBOSE
dverb_info[0] = (int)rb->idnumber;
dverb_info[1] = rb->alfa;
dverb_info[2] = rb->beta;
dverb_info[3] = rb->ply; // depthleft
DVERB(4,dverb_abigetblock);
#endif
Now the function DVERB is doing this :
void DVERB(int ninfo,int dverbflag) {
int header,i;
/* header byte:
* 32 bits waarvan:
* 1 sign (unused)
* 17 laatste debugsmp punt (indien aanwezig)
* 10 verbosepoint: dverb_...
* 4 processornummer (max 0..15)
*/
//printf("locking dverb ninfo=%i flag=%i\n",ninfo,dverbflag);
Lock(tree->lock_dva);
header = 0;
//#endif
header |= (dverbflag<<4)|ProcessNumber;
dva[tree->dva_count++] = header;
i = 0;
while( i < ninfo ) {
dva[tree->dva_count++] = dverb_info[i];
i++;
}
if( tree->dva_count >= DVASIZE-64 ) {
int tdr;
printf("het probleem is duidelijk!\n");
tdr = tree->dva_rond+1;
// printf("Passed DVASIZE (%i bytes) already %i
times!\n",sizeof(int)*tree->dva_count,tdr);
if( nprocesses == 1 && false ) {
//printf("gonna save it now\n");
Unlock(tree->lock_dva);
SMPdebugSAVE();
InitVerbose();
Lock(tree->lock_dva);
}
else {
int m,move;
//mprint("processes = %i rounds = %i\n",tree->nprocesses,tdr);
InitVerbose();
/* nu vastleggen het volgende: de zetstack van de verschillende processors
*/
dva[tree->dva_count++] = (dverb_movestacks<<4);
for( i = 0; i < nprocesses; i++ ) {
dva[tree->dva_count++] = 5;
m = 0;
//mprint("stored line in DVA: %i ... ",tree->movesmadeindex[i]);
while( m < 5 ) {
move = m;
mprint("%s%s ",M2A[(move>>6)&63],M2A[move&63]);
dva[tree->dva_count++] = move;
m++;
}
//mprint("\n");
}
}
tree->dva_rond = tdr;
}
Unlock(tree->lock_dva);
}
So basically lock, store the search info as integers in the array and unlock.
When it has crashed, all you have to do is write the shared memory to disk and
then parse the shared memory until you found the processor that crashed and you
figure out where it was and what it was doing and the state at which other
processors were at that moment and so on.
And then pray that you crash very quickly. Basically when this testbed works
fine all you have got to do is find the quickest way to crash or get a deadlock
or race condition in your program. Then debug backwards.
That's not so easy and requires a lot of experience :)
But you can get very complex parallel forms of search to work with this method.
Because basically every processor directly locks and unlocks, the odds are not
so big that this 'debug code' in shared memory will eat all system time.
Your evaluation will, your move ordering will and so on.
It's just a sequential fill of shared memory.
Then additional write tools to find relevant data within that huge file.
Ways to really quickly crash your program is important to find too. A good one
i'll give for free, that's use the utmost stupid way to split, like splitting
always and especially as much near the leaves as possible. And start searching
with 16 cpu's :)
That really crashes the program quickly :)
Now the way in which i print out things. To give a real example of 16 processors
(note my printing routines aren't even good working for it) :
0............... SplitInOthers: processnumber=0 newproc=8 idnumberposition=1236
0............... SplitInOthers: processnumber=0 newproc=15 idnumberposition=123
6
0............... SplitInOthers: processnumber=0 newproc=1 idnumberposition=1236
..........10..... Splitting position id=1236
..........10..... Splitting position: found and moved to position 1236
.......7........ Splitting position id=1236
.........9...... Splitting position id=1236
.........9...... Splitting position: found and moved to position 1236
.......7........ Splitting position: found and moved to position 1236
.....5.......... Splitting position id=1236
.....5.......... Splitting position: found and moved to position 1236
..2............. Splitting position id=1236
..2............. Splitting position: found and moved to position 1236
...........11.... Splitting position id=1236
...........11.... Splitting position: found and moved to position 1236
........8....... Splitting position id=1236
........8....... Splitting position: found and moved to position 1236
............12... Splitting position id=1236
............12... Splitting position: found and moved to position 1236
...3............ Splitting position id=1236
...3............ Splitting position: found and moved to position 1236
......6......... Splitting position id=1236
......6......... Splitting position: found and moved to position 1236
....4........... Splitting position id=1236
.............13.. Splitting position id=1236
..............14. Splitting position id=1236
....4........... Splitting position: found and moved to position 1236
...............15 Splitting position id=1236
.1.............. Splitting position id=1236
..............14. Splitting position: found and moved to position 1236
...............15 Splitting position: found and moved to position 1236
.............13.. Splitting position: found and moved to position 1236
.1.............. Splitting position: found and moved to position 1236
............12... Abortme() id=1236
............12... splitid=0 depth=5 procs=[0,6,10,3,2,-1,9,13,4,11,5,7,14,8,15,
1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
............12... PartBoth: proc just returned to idle searching 0 nodes
............12... ActionCheck() flags=1 realply=0
.....5.......... Abortme() id=1236
.....5.......... splitid=0 depth=5 procs=[0,6,10,3,2,-1,9,13,4,11,-1,7,14,8,15,
1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
.....5.......... PartBoth: proc just returned to idle searching 0 nodes
.....5.......... ActionCheck() flags=1 realply=0
..............14. Abortme() id=1236
..............14. splitid=0 depth=5 procs=[0,6,10,3,2,-1,9,13,4,11,-1,7,-1,8,15
,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
..............14. PartBoth: proc just returned to idle searching 0 nodes
..............14. ActionCheck() flags=1 realply=0
...3............ Abortme() id=1236
...3............ splitid=0 depth=5 procs=[0,6,10,-1,2,-1,9,13,4,11,-1,7,-1,8,15
,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
...3............ PartBoth: proc just returned to idle searching 0 nodes
...3............ ActionCheck() flags=1 realply=0
......6......... Abortme() id=1236
......6......... splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,13,4,11,-1,7,-1,8,1
5,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
......6......... PartBoth: proc just returned to idle searching 0 nodes
......6......... ActionCheck() flags=1 realply=0
.............13.. Abortme() id=1236
.............13.. splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,4,11,-1,7,-1,8,
15,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
.............13.. PartBoth: proc just returned to idle searching 0 nodes
.............13.. ActionCheck() flags=1 realply=0
....4........... Abortme() id=1236
....4........... splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8,
15,1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
....4........... PartBoth: proc just returned to idle searching 0 nodes
....4........... ActionCheck() flags=1 realply=0
.1.............. Abortme() id=1236
.1.............. splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8,
15,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
.1.............. PartBoth: proc just returned to idle searching 0 nodes
.1.............. ActionCheck() flags=1 realply=0
...............15 Abortme() id=1236
...............15 splitid=0 depth=5 procs=[0,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8
,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
...............15 PartBoth: proc just returned to idle searching 0 nodes
...............15 ActionCheck() flags=1 realply=0
0............... Abortme() id=1236
0............... splitid=0 depth=5 procs=[-1,-1,10,-1,2,-1,9,-1,-1,11,-1,7,-1,8
,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
0............... PartBoth: proc just returned to idle searching 0 nodes
0............... ActionCheck() flags=1 realply=0
..........10..... Abortme() id=1236
..........10..... splitid=0 depth=5 procs=[-1,-1,-1,-1,2,-1,9,-1,-1,11,-1,7,-1,
8,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
..........10..... PartBoth: proc just returned to idle searching 0 nodes
..........10..... ActionCheck() flags=1 realply=0
..2............. Abortme() id=1236
..2............. splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,11,-1,7,-1,
8,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
..2............. ActionCheck() flags=1 realply=0
.......7........ Abortme() id=1236
.......7........ splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,11,-1,-1,-1
,8,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
.......7........ PartBoth: proc just returned to idle searching 0 nodes
.......7........ ActionCheck() flags=1 realply=0
........8....... Abortme() id=1236
........8....... splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,11,-1,-1,-1
,-1,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
........8....... PartBoth: proc just returned to idle searching 0 nodes
........8....... ActionCheck() flags=1 realply=0
...........11.... Abortme() id=1236
...........11.... splitid=0 depth=5 procs=[-1,-1,-1,-1,-1,-1,9,-1,-1,-1,-1,-1,-
1,-1,-1,-1] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
...........11.... PartBoth: proc just returned to idle searching 0 nodes
...........11.... ActionCheck() flags=1 realply=0
.........9...... MakeSplittable() id=1233 realply=3
.........9...... registeridlesplits owner proc=9
.........9...... splitid=0 depth=3 procs=[11,8,7,2,10,0,9,15,1,4,13,6,3,14,5,12
] daughters=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
.........9...... SplitInOthers: processnumber=9 newproc=11 idnumberposition=123
3
and so on
As you see this output is just printing all actions about parallel search, not
even printing info regarding the actual searched tree (as that would be hundreds
of megabytes).
The huge advantage of writing the debug info to shared memory is that it eats so
little time that the processors get more time into their splitting routines and
get therefore bigger odds to crash if there is a bug.
>>Apart from boundschecking with a real boundschecker; gcc has a great
>>boundschecker extension at their homepage, windows has nothing that impressed me
>>in fact like devpartner doesn't even find the most obvious boundsbug in my GUI.
>>Further there is normal debugging ways, the most effective way in which i debug
>>Diep if there is problems is in the next trivial way:
>>
>>That simply printing the entire search tree to disk. That can be done either
>>direct or indirect :
>>
>>example output (written by hand just to show example now for 4 processors) :
>>
>>0... searching root
>>0... makemove a2a3
>>0... a2a3 makemove d7d5
>>0... a2a3 d7d5 eval -0.200
>>0... a2a3 d7d5 ab=[0.001;0.002] cutoff -0.200
>>...
>>
>>And when processor 1 is doing an action :
>>.1.. blabla
>>
>>In short this is a very powerful way to debug search or debug parallel bugs.
>>Of course for tough race conditions in a parallel search it is not so clever to
>>directly print to textfile, as the time spent to print is too long (which means
>>a deadlock or race conditions will happen not so soon as the time spent to
>>printing is too huge).
>>
>>So i print it in a shared block of memory (locking first of course) and later
>>parse that memory block to the above text.
>>
>>In this way i managed to debug the parallel search of Diep.
>>
>>For single cpu just printing directly to disk will do fine and you can examine
>>how big your trees get at small search depths up to a ply or 6 is no problem.
>>
>>You'll find many weird stuff in your qsearch and your move ordering like this.
>>
>>Of course it's time consuming to do the above (reading those huge textfiles) but
>>it's very rewarding.
>>
>>The above has another huge advantage and that it is finding bugs in your
>>evaluation function and directly gives you an answer whether a new compiler
>>sucks, is dangerous to use or whether you have a so far unfound bug which
>>boundscheckers didn't find.
>>
>>That is, provided you also print the evaluation score in the textfile.
>>
>>You can compile at different platforms with different type of compilers your
>>code and just compare the textfile verbose. If it is not 100% similar, then
>>obviously there is a bug somewhere.
>>
>>This really will remove about all nasty to find bugs in your entire
>>search&evaluation function.
>>
>>It is the by far the best test to see whether a new compiler has bugs.
>>
>>Especially many intel compilers and certain gcc compilers did not pass this test
>>in past when using -O3 or in case of intel just -O2. gcc 3.3.x and 3.4.x was
>>real real buggy here. 4.0 is far better.
>>
>>Of course the fact that you *seriously* look to debugging methods in your engine
>>already means that your thing is going to be more bugfree than majority of
>>amateur programs. Most haven't heard in fact from the word 'boundschecker'.
>>
>>The fact that out of any random compiler with this trick i can find
>>optimizations that produce bugs, tells me how poor the different compiler teams
>>are in debugging their compilers.
>>
>>Diep is not using any difficult casting construction or so, i never take risks
>>there. Additional it is 100% integer code, so no floating point tricks are
>>possible for compilers.
>>
>>Especially the GCC team is the most horrible code writing team on planet earth.
>>
>>Like 99% of their releases suffers so much from bugs that i can't even compile
>>with -O3 without catching bugs.
>>
>>Obviously that i can prove that with such a simple debugging method is telling
>>really a lot about how powerful the verbose printing method of a search is.
>>
>>>I was wondering if anyone would like to volunteer any tricks they've
>>>used to help find certain bugs in their search function.
>>
>>>I think that the ideas of using perft for move generation and
>>>reversing the board to find bugs in the evaluation have both
>>>been really useful to me. I was wondering if anyone has used
>>>techniques similar to these to help find search bugs.
>>>
>>>I understand that just because a engine can properly pass these
>>>and other tests doesn't mean it's free of bugs. But they certainly
>>>help detect at least some of them.
>>>
>>>I'm certain that there must be plenty of bugs in Latista's search
>>>and I think it's time for me to work on discovering them. If
>>>you don't have any automated tricks like above. Does anyone
>>>have any general advise to help me spot some bugs?
>>>
>>>Eric Oldre
>>>
>>>PS. I have at various spots in my program tried to follow a similar
>>>model of asserts as in fruit. I'm sure taking some time to
>>>do this at more parts of my program would help.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.