Author: Robert Hyatt
Date: 11:26:42 10/07/99
Go up one level in this thread
On October 07, 1999 at 12:23:36, Didzis Cirulis wrote: > >> >>I think that the original tester of this long thread Dizdis cirulis played >>clearly more slower games and manually. >> >>Of course the experiment is not the same as with 2 computers but I think that >>the difference is usually minor(less than 20 elo). >> >>I believe that Christophe Theron may improve his program by tuning to permanent >>brain on(he admitted that he does not do it) but I do not think that the >>improvement is going to be more than 20 elo. >> >>You also can improve your program for permanent brain off games >>by tuning it to this kind of game but I do not believe that you will get more >>than 20 elo realtive to a good guess without tuning. >> >>Uri > >Time for me to show up again ;-) > >When testing on one computer with PB off, there are not so much problems at all. >If Crafty does some more work after the actual move, no problem - when move is >made, crafty may do what ever it wants in no hurry. If no permanent brine acting >in this, no problem. So, crafty makes a move, updates learning etc and.... and >stands still until the next move. Right or wrong? And even if it would do >something (like CM) in background, I would see the slowdown in other program's >work. And I would correct times for the game. >The same for CM - That does not matter if I turn that dialogue window on >immediately! That is not CM's PB that is wasting time. And CM's opponent is >"sleeping" there in background not taking care about what CM does until I come >to feed in the CM's move. AT THIS MOMENT when I am about to feed that move into >CM's opponent, CM itself must be put to "sleep" by starting that dialogue >window. What is so difficult about understanding this? One program works while >another is stopped. No uncontrolled influence! > >Now about that "cripled game mode" argument: I agree there is something >importand that differs - PB is off. But let me point to another aspect of this: >My main testing is for Chess Tiger. And whatever "cripled" data I produce, >Christophe is clever enough to use it to improve his program. Is there any doubt >about Chess Tiger's progress? > >So, conclusion: > >1) I am happy to play my test games as that is fun. I learn myself a lot while >doing that - trying to guess the upcoming move etc. > >2) Christophe is happy as his program is getting better. > >3) I fell good as I take a part in computer chess progress ;-) by helping >Christophe. > >This is all an offtopic (maybe) for Programmers, but sometimes I read your >messages and I feel you miss something important. Correct me if I am wrong. > >Have a great Chess! > >Didzis Cirulis There are two aspects to this sort of testing: (1) yes, games with PB off are useful to the programmer. Because you can _still_ pick up on weaknesses in the evaluation, in the search, quirks that make it play bad moves, etc. (2) no, games with PB off are not comparable with games with PB on. The results can be significantly different. I usually try to play many games manually before a WMCCC event, at the actual time control being used, so that I can monitor time usage, and see how it behaves in most circumstances. Such games with ponder=off would _not_ help me one iota in preparing for these events. yes, I would see if it makes bad moves or good moves, or misses tactics, or finds tactics. But I wouldn't see anything about how it is going to allocate its time, how it will arrive at the time control with how much time left, and so forth. It is (to me) amazing how many crafty vs X games get reported here where (a) crafty does great, because "X" is not set up properly; (b) crafty does poorly because crafty is not set up properly; (c) crafty does strangely because the default evaluations are not being used; (d) crafty gets smashed because it only plays gambits due to some odd opening book being used; (e) crafty smashes the opponent because _it_ is using some non-optimal opening book. I would like to see results reported _when_ _they_ _mean_ _something_. I hate to spend time going thru games just to find "jeez what a lemon book move, no wonder it won or lost" game after game. Or "Jeez, no way can I make it reproduce this move" and then discover it was a 'modified' crafty. Or "Jeez this was a bad move" only to find that at move 40 it had almost no time left becuase the game was played with ponder=off and it screwed up allocating time somewhere along the way... I play lots of ponder=off matches here vs other programs, because when I find something fishy going on, I can reproduce the games easily, while with ponder=on (which would be fine on my multiple-cpu box) makes it hard or impossible to reproduce a specific game. But you notice that I don't report the results of those ponder=off games any more than I would report results of matches played on two different types of machines or whatever. They are difficult to interpret...
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.