Author: Andrew Wagner
Date: 13:53:59 05/26/04
Go up one level in this thread
On May 26, 2004 at 01:37:00, Will Singleton wrote:
>On May 25, 2004 at 23:09:16, Andrew Wagner wrote:
>
>>On May 25, 2004 at 22:37:58, Will Singleton wrote:
>>
>>>On May 25, 2004 at 22:01:31, Andrew Wagner wrote:
>>>
>>>>On May 25, 2004 at 21:00:52, Will Singleton wrote:
>>>>
>>>>>On May 25, 2004 at 20:09:57, Andrew Wagner wrote:
>>>>>
>>>>>>On May 25, 2004 at 19:36:04, Will Singleton wrote:
>>>>>>
>>>>>>>On May 25, 2004 at 17:44:55, Andrew Wagner wrote:
>>>>>>>
>>>>>>>>I do a lot of reading through CCC archives. I use the search engine from here,
>>>>>>>>and also I'm in the process of reading through the old archives systematically
>>>>>>>>using the offline reader (I'm in the fall of 2001 currently, I think). Anyway,
>>>>>>>>sometimes I run across a nugget that makes me just stop and go "whoah". Here's a
>>>>>>>>quote from one of Bob's posts, originally about hashing algorithms:
>>>>>>>>
>>>>>>>>>I think the key to improving a program, once it plays legally, is to develop
>>>>>>>>>a methodology to carefully profile the code, find the hot spots, and then find
>>>>>>>>>ways to speed up those hot spots. But all the while paying _careful_ attention
>>>>>>>>>to the overall node counts on a wide range of test positions. A 1% speedup is
>>>>>>>>>of no use at all if you introduce an error that happens once every billion
>>>>>>>>>nodes. I can search that many nodes in 15 minutes. I can't stand errors that
>>>>>>>>>frequently. I have what would probably be called a "zero-tolerance for errors"
>>>>>>>>>in Crafty. If I make a change that should only make it faster or slower, then
>>>>>>>>>the node counts must remain constant. If they don't I debug until I find out >why and fix it.
>>>>>>>>
>>>>>>>>This is a fantastic point. Maybe somewhat obvious to our more experienced
>>>>>>>>members, but certainly words of wisdom for us newbies. So, my question is, what
>>>>>>>>methods are you all using for profiling your code? How do you go about
>>>>>>>>identifying and fixing your hotspots? Do you have a particular test suite you
>>>>>>>>use, or what? Andrew
>>>>>>>
>>>>>>>I'm surprised Bob would say that profiling is important so soon in the
>>>>>>>development process; perhaps there's some missing context. Profiling is, imho,
>>>>>>>about the last thing you'd want to do.
>>>>>>
>>>>>>Here's the link, so you can read it in context, if you'd like:
>>>>>>http://chessprogramming.org/cccsearch/ccc.php?art_id=112972
>>>>>>
>>>>>>>
>>>>>>>1. Fix bugs in movegen, using perft tests.
>>>>>>
>>>>>>He does say "once it plays legally". To me, that implies a bugfree movegen...
>>>>>>
>>>>>>>2. Write a very simple, bug-free eval.
>>>>>>>3. Concentrate on move-ordering, which is crucial to making the tree small.
>>>>>>>Develop methods for measuring the quality of your ordering, don't only look at
>>>>>>>node counts.
>>>>>>>
>>>>>>
>>>>>>I would count these as what he calls "hot spots". Especially move ordering
>>>>>>(though good eval helps move ordering).
>>>>>>
>>>>>>>Don't spend a lot of time on arcane or new ideas until you're certain what you
>>>>>>>have is bug-free. Especially make sure your transposition code is simple and
>>>>>>>effective, tons of problems result from bad hashing.
>>>>>>
>>>>>>I would also be interested in a process for this. What process do you use to
>>>>>>really be absolutely sure your program is bug free? Especially your hash table
>>>>>>code. E.g. at the moment, I think Trueno is bug free, more or less. But I
>>>>>>haven't found a good thorough test to give it that will tell me.
>>>>>>
>>>>>>>
>>>>>>>Once you have a good, stable platform to build on, you can be sure that your
>>>>>>>future experimentation will be productive.
>>>>>>
>>>>>>But what does the "build on" process consist of? That's the question. Bob's
>>>>>>answer is this profile/find hotspots/fix-while-watching-node-counts process. But
>>>>>>how do you implement this?
>>>>>
>>>>>It's a good area for discussion. By way of background, a question: there are a
>>>>>few starter programs which seem to be debugged and simple, like gerbil and tscp.
>>>>> How does Trueno do against those?
>>>>
>>>>Trueno usually gives TSCP a run for its money, but I don't think it's managed
>>>>better than a draw yet. I haven't tried it with gerbil...I can't seem to get
>>>>gerbil working with winboard. I'll have to fool with that some more.
>>>
>>>I just need to be clear, do you mean Trueno usually gets 50% or less in a match
>>>against tscp, or do you mean Trueno hasn't yet beaten tscp in a game?
>>
>>Ooooh, you're gonna make me say it aren't you? Ok, ok, as far as I can remember
>>Trueno has not yet EVER beaten TSCP in a game. Now, to soothe my pride, let me
>>add that I got Gerbil working, and Trueno just won this interesting little 30 0
>>game against it:
>>
>>[Event "30 0 game"]
>>[Site "?"]
>>[Date "2004.05.25"]
>>[Round "?"]
>>[White "Trueno"]
>>[Black "Gerbil"]
>>[Result "1-0"]
>>[Opening "King's Indian: Andersson variation"]
>>[ECO "E92"]
>>[NIC "KI.18"]
>>[Time "23:01:01"]
>>
>>1. d4 Nf6 2. c4 g6 3. Nc3 Bg7 4. e4 d6 5. Nf3 O-O 6. Be2 e5 7. dxe5 dxe5 8.
>>Be3 Qxd1+ 9. Rxd1 Bg4 10. h3 Bxf3 11. Bxf3 Rc8 12. h4 c5 13. h5 gxh5 14. Nd5
>>Nxd5 15. cxd5 h4 16. Bg4 Re8 17. Bxc5 Na6 18. Ba3 Bf6 19. Rd3 Red8 20. Rf3
>>Kg7 21. Rb3 b6 22. Rc3 b5 23. Rb3 Rab8 24. Be2 Rdc8 25. Rc3 b4 26. Rxc8 Rxc8
>>27. Bxa6 Rc1+ 28. Ke2 Rxh1 29. Bxb4 h3 30. gxh3 Rxh3 31. Bc5 Rh4 32. Bd3 a5
>>33. Bb6 Kf8 34. Bxa5 Ke7 35. a4 h5 36. Bb4+ Kd7 37. Bc5 Kd8 38. b4 Rh2 39.
>>d6 Rh4 40. a5 Kc8 41. a6 Bg5 42. d7+ Kc7 43. Bd6+ Kc6 44. b5+ Kxd7 45. a7
>>Kxd6 46. a8=Q Rg4 47. f3 Rg2+ 48. Kf1 Rd2 49. Qd5+ Kc7 50. Qxf7+ Kb6 51.
>>Qg6+ Ka5 52. Qxg5 Rxd3 53. Ke2 Rb3 54. Qxe5 Rxb5 55. Qa1+ Kb6 56. f4 Ra5 57.
>>Qc3 Rc5 58. Qb2+ Rb5 59. Qd4+ Ka5 60. Qh8 Ka4 61. f5 Rb4 62. Kd3 Rb3+ 63.
>>Kc4 Rb4+ 64. Kd5 Rb5+ 65. Kc6 Rb1 66. Qxh5 Kb4 67. f6 Rc1+ 68. Kd6 Rc8 69.
>>e5 Rd8+ 70. Ke7 Rb8 71. Qh1 Kc5 72. f7 {Black resigns} 1-0
>>
>>I should also point out that I have not done a lot of testing with these
>>engines, because my arena doesn't seem to be able to run tournaments correctly.
>>If it did, I'd be running a lot more of them, for testing purposes. As it is,
>>all the tests I do get done by hand, one game at a time, with winboard.
>
>Make a shortcut to winboard, and put this in the shortcut target window:
>
>C:\chess\winboard.exe -cp /xponder /tc 5 /mps 40 /inc -1 /mg 50 /sgf games.pgn
>/fcp=amateur280 /fd="c:\chess\amateur280" /scp=trueno /sd="c:\chess\trueno"
>
>substituting paths and engine names, of course. The mg option will play, in
>this case, 50 games.
>
>If you're having trouble with tscp, then I submit you probably have either some
>bugs or else a faulty search design. You must get that ironed out before going
>on.
>
>In the gerbil game, gerbil made some odd moves. For example, 15...h4. My copy
>of gerbil discards that at 6 seconds (p4 2.8ghz), so perhaps you are using a
>slower machine. Your move 25.Rc3 shows a lack of search depth (capture
>extensions?), or perhaps a qsearch problem.
>
>I don't know, Andrew. Finding bugs is hard, but it's essential. You shouldn't
>waste time trying advanced crap unless you have a solid base to work from. Take
>it from me, I've wasted a lot of time, more than I care to say!
>
>To debug, there's so many things you can do. Using conditional compiler
>directives, you can test every aspect of your program, and print to a log any
>inconsistences or unexpected behavior you find. This error-testing takes a long
>time to do, but it will pay off in the long run.
Ok Will, thanks for the interesting discussion. I just matched up Trueno with
TSCP for 30 games. The result was NOT pretty. So...I'm going to have Crafty go
through and annotate the games with a nice large blunder margin, so I can figure
out what moves are costing me the games. This way I can debug from the positions
where I'm throwing away games and see what jumps out. We'll see what happens!
Andrew
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.