Author: Robert Hyatt
Date: 07:27:55 10/07/99
Go up one level in this thread
On October 07, 1999 at 00:45:55, Christophe Theron wrote: >On October 06, 1999 at 00:50:40, Robert Hyatt wrote: > >>On October 06, 1999 at 00:40:23, Christophe Theron wrote: >> >>> >>>Probably because you feel like you are allowed to make rock statements like >>> >>> "Testing on _one_ machine is a bad practice, period." >>> >>>and that you do that everytime you post here, as if you were the ultimate expert >>>here. >>> >>> >> >> >>Err... I believe I _am_ the "resident expert" when it comes to Crafty, wouldn't >>you think? IE who would claim to know more about how it works, how it was >>designed, why it was designed that way? >> >>And whether you like it or not, one more time: >> >> "using crafty, testing on one machine _is_ a bad idea. If you don't like >> that, tough. test your own program in any way you want. I don't want to >> take the time to fiddle with this." > > >I prefer this sentence. State clearly that what you say apply to Crafty only. > No problem. But note that Ed has said the same thing about Rebel. Note that Frans has said "give fritz all the hash you can". So several believe that testing on one machine is bad for various reasons. I can cite module and line# for crafty showing where problems occur. I assume Ed and Frans can do the same. no doubt others would find the same things if they looked at their programs as well... > > >>>> but I'm up to the task. >>> >>> >>>I know. >>> >>> >>> >>>>If it takes a 'crippled' machine for you to find new algorithms or find old >>>>bugs, fine. I have a better testing methodology. Because such testing is not >>>>exactly a modern approach to developing software. It is just as easy to cut the >>>>search time by 90% rather than finding a machine 1/10th the speed. Exactly the >>>>same result... >>> >>>No. >>> >>>I'm using the standard PC timer resolution, which is about 0.05 seconds, to make >>>time measurements in my program. >>> >>>My K6-300 is about 100 times faster than my 386sx20. And I wanted, for some >>>purpose, to play blitz games at 386sx20 rate. >>> >>>To get on the K6 the same time per move, I should have set it to game in 3 >>>seconds, which makes something like 0.05 seconds per move in average. This is >>>too close from the PC timer resolution, so I had to test on 386. >>> >>>The purpose of the test was to check if my pruning near the horizon was as good >>>as the pruning of very old programs (Psion and Genius to name them). It was not, >>>and if I had no 386 at hand I would probably have overlooked it. >>> >>>I would not run on a 386 all day, but in this case it helped. >>> >>> >>> >>> >>>>>As for "will it be useful in serious games", keep an eye on the results of Tiger >>>>>and you will know. >>>>> >>>>>Thanks for your time and patience, Doctor Hyatt. >>>>> >>>>> >>>>> >>>>> Christophe >>>> >>>> >>>>I'll bet you a buck that your testing with ponder=off won't help you one iota >>>>in "real" games... Because you won't be playing with ponder=off in "real" >>>>games, just like the rest of us.. >>> >>>I *never* play real games, except when I operate Tiger at a tournament (or the >>>day before to be sure that playing real games still works). >>> >>>I have *never* played a long time control game (40 moves in 2 hours) in my "lab" >>>since more than 5 years. >>> >>>The last time I played a game in 1 hour in the lab was 2 years ago. 2 games >>>against CM4000, I remember. >>> >>>99% of the games I play with Tiger are on a single computer (switching back and >>>forth between the 2 programs under Windows), and of course PB off. >>> >>>So if testing with PB off does not help me, I guess you can deduce I have made >>>no progress since more than 5 years. >>> >>> >>> Christophe >> >> >>Please try to read carefully. testing _specifically_ to tune the ponder=off >>mode of playing will _not_ help you win a game anywhere, except when you play >>ponder=off games. > > >Well... My time management code has been designed at first for PB off games (as >I didn't have PB at that time), and it works very well without any change for PB >on... No reason that couldn't happen. My program had "pondering" from day 1, in the form of Crafty, so that is the only way I have ever tested... > >Probably it's just because I have another (simpler) philosophy about how to use >the time I save in PB. > possibly. Bert Gower and I spent years studying how programs (and GM players) use their time during a game. I don't like 'linear increasing time allocation' at all, as I don't want / need to take 10 mins a move for a KP vs K ending. I'd rather use the time to make sure I am the one with the P when I get there. > > >> Testing with ponder=off is fine. I do it all the time to >>get more repeatibility when testing or debugging. > > >Okay. I do the same. > > > >> But I don't test/tune >>_specifically_ for that mode (ponder=off). As a result, that mode is much less >>tested than ponder=on. I have seen the result. And I don't like what I see, >>and I don't have the time, nor do I want to take the time to improve that which >>has nothing to do with 'winning real games.' > > >It took me no time at all. > > > >>end of story... >> >>cute/snide remarks don't cut bait here. If you want to make it that when I >>tell someone that my program isn't set up to work in a mode they are using it, >>that I am calling them stupid... or if you want to twist what I wrote to say >>you have made no progress... that's your choice. But it isn't what I _said_. > > >You have implied repeatedly that testers don't control what's happening right in >front of them and that they are probably unable to give accurate results when >they test on one computer only. I have implied that. And I have good reason to do so. I explained my 'learning cycle' that burns some cpu and some I/O at _one_ point in the middle of a game. no one has mentioned this. But in a game/1 minute tournament that will absolutely affect one move of the opponent in every game. Yet not one person has noticed/mentioned this. So there appears to be good reason to suspect that the testers are not as observant as you are thinking. Not to mention all the other bad things about using one machine... shortchanging fritz on hash memory, sharing books, etc. > >You have made your point clear that Crafty doesn't behave correctly when >ponder=off. Very well. You didn't have to spit the kind of statements that are >BTW usual from you Not sure what you mean. I believe _you_ started the somewhat ugly tenor this thread took. I simply supplied facts, and examples, of why this sort of testing is bogus. You started claiming that I was calling the testers stupid and so forth, which I didn't do. Doing something wrong because of not knowing better or not understanding the two programs doesn't mean 'stupid'. Stupid and ignorant are not the same thing at all. I am ignorant about a _lot_ of things. But computer chess doesn't happen to be one of them.. > > "Testing on _one_ machine is a bad practice, period." > >What I like probably the most here is "period". > >It says: "don't bother me with poor testers that cannot even afford 2 >computers". It doesn't say that at all. It says that testing on _one_ machine is a bad test. Nothing else. IE I have several friends that are physicists... I'd hardly say "don't bother me with poor testers than can't afford a particle accelerator." That isn't the issue. If the experiment requires an accelerator, then either use one or don't do the experiment. Because using something sub-optimal is not always 'good science'. one machine testing fits this metaphor quite well. It works. But that doesn't mean it is good science to do it. > >From other discussions we had before I guess that: > >* if you don't have 2 computers >* if they are not very fast >and >* if they don't have 64 bits processors Don't know where you get that last one from. Point (a) above is correct. Point (b) is correct to an extent, as on a 486 Crafty will die due to null- move failures in the shallow searches. So long as people have used something like a pentium-pro 200mhz or faster, things seem to work well and I have said that. Alphas are far better. But I don't use alphas myself, if you'll notice, I am running on intel 100% of the time here... > >THEN WHY THE HELL ARE YOU USING CRAFTY? IT IS _BAD_ PRACTICE, PERIOD. can you spell hyperbole? Now look it up... > >* It is not designed to play a game with ponder=off true... >* It is not designed for slow computers true... >* It is not designed for 32 bits computers. > true, but it plays just fine on 32 bit computers... > >Am I correct, or is one of the above points wrong? > depends on the point you make with the point. It is designed around 64 bit machines. It plays fine on decent 32 bit machines. Which is why I test only on my PII/xeon... > > > Christophe
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.