Author: Robert Hyatt
Date: 05:26:35 10/08/99
Go up one level in this thread
On October 08, 1999 at 03:26:39, Peter Kappler wrote: >On October 07, 1999 at 14:12:53, Robert Hyatt wrote: > >>On October 07, 1999 at 06:07:49, Ed Schröder wrote: >> >>>>Posted by Robert Hyatt on October 06, 1999 at 15:30:48: >>>> >>>>I'm going to make a list of all the reasons why two programs, one computer, >>>>is a bad thing to do: >>>> >>>>1. a program might not be well-adjusted in how it uses its time when it is >>>>not allowed to "ponder". Crafty is an example. >>>> >>>>2. a program might not be well-behaved and do some unexpected computation >>>>after it sends the move to the referee program. IE in crafty, I send the move, >>>>then I do the learning stuff after 10 non-book moves have expired. This 'learning >>>>cycle' can take 2-3-4 seconds with a really large book and a long opening line >>>>in the book. Imagine what that does to a game/1minute time control that many >>>>are using in winboard/xboard? >>>> >>>>3. a program (ie crafty) might do other things after it annouces its move, >>>>such as malloc()'ing a large buffer for (say) learning or whatever. What does a >>>>large malloc() do to the other program? swap it out? >>> >>>Never thought of this. This is a real killer, I agree. If done on purpose you >>>win every game :-) >>> >>>>4. A program (ie chessmaster) might poll for input, consuming 1/2 of the cpu >>>>even though it is not 'thinking'. >>> >>>Another true point. Every program needs to poll for input. Asking the >>>keyboard / mouse for input are expensive (slow) operations. In Rebel >>>I have a counter that makes sure to look for input after 500 evaluations. >>>If I decrease the value to say 50 or 10 the NPS of Rebel drops >>>tremendously (forgot about the exact slow-down). >>> >>>Thus the opponent program will ALWAYS slow-down yours and you can >>>only guess how much that is. >>> >>>>There are _too_ many things a program _might_ do. I'll bet not one person >>>>gave any thought to a "learning cycle" in crafty, yet it does it in every >>>>game. And it steals 2-4 seconds of time from the opponent. In short time >>>>controls, that might be important. >>>> >>>>If I _know_ people are testing like this, I'll bet I can raise Crafty's rating >>>>by 100 points minimum. I won't say how, but it shouldn't take too much >>>>imagination to figure it out. :) And with that said, why bother testing in a >>>>way that is obviously potentially unreliable. For fun, sure. But reporting >>>>the results as "A beats B" is not very scientific... A might not actually >>>>be able to beat B, he just might have a smarter programmer that takes >>>>advantage of a flawed testing methodology... >>>> >>>>:) >>>> >>>>Bob >>> >>>Strong points Bob. Still people are in love with the system as you only >>>need one PC and have a lot of fun. If only its results are judged in the >>>way it should. And in no way you can compare eng-eng matches on 1 PC with >>>eng-eng matches on 2 PC's. >>> >>>Ed >>> >>>PS, match score sofar PB_ON vs PB_OFF 17.5 - 8.5 >> >> >>I suspect you are going to get a 100+ point difference. When I first started >>running Crafty on ICC, I was experiencing the usual "scanf()" buffering issue >>with winboard and couldn't get pondering to work right, although it was fine >>in text mode without xboard. When I got it right, crafty's rating instantly >>jumped by about 120 rating points, maybe more. When I would turn it off, down >>it would go by the same amount. I have, as a result, generally credited >>ponder=on with roughly 120 rating points. Be interesting to see how your >>results end. You are at about the 120 point level yourself, so far, if my >>mental calculations are anywhere near right, winning 2 of every 3 games. 3 of >>every 4 gets to +200, so maybe you are closer to 160 than 120... we will see... > > >It's worth pointing out that whatever rating difference he gets is going to be >inflated, since he is using the same engine on both machines and therefore >getting perfect ponder accuracy. > >--Peter It won't get 'perfect ponder accuracy' by a long shot. I doubt he will predict over 50%. Because one engine is _always_ able to outsearch the other by 1 ply on its turn. because both will be doing about the same depth normally, but when one does an 8 ply search, it only has info based on a 7 ply search by its opponent. When its opponent actually moves it will have 8 plies (or whatever) to choose a move. Pondering accuracy won't be nearly as high as you think... particularly when the game starts to turn tactical.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.