Author: Robert Hyatt
Date: 14:29:52 10/08/99
Go up one level in this thread
On October 08, 1999 at 12:37:45, blass uri wrote: >On October 08, 1999 at 08:26:35, Robert Hyatt wrote: > >>On October 08, 1999 at 03:26:39, Peter Kappler wrote: >> >>>On October 07, 1999 at 14:12:53, Robert Hyatt wrote: >>> >>>>On October 07, 1999 at 06:07:49, Ed Schröder wrote: >>>> >>>>>>Posted by Robert Hyatt on October 06, 1999 at 15:30:48: >>>>>> >>>>>>I'm going to make a list of all the reasons why two programs, one computer, >>>>>>is a bad thing to do: >>>>>> >>>>>>1. a program might not be well-adjusted in how it uses its time when it is >>>>>>not allowed to "ponder". Crafty is an example. >>>>>> >>>>>>2. a program might not be well-behaved and do some unexpected computation >>>>>>after it sends the move to the referee program. IE in crafty, I send the move, >>>>>>then I do the learning stuff after 10 non-book moves have expired. This 'learning >>>>>>cycle' can take 2-3-4 seconds with a really large book and a long opening line >>>>>>in the book. Imagine what that does to a game/1minute time control that many >>>>>>are using in winboard/xboard? >>>>>> >>>>>>3. a program (ie crafty) might do other things after it annouces its move, >>>>>>such as malloc()'ing a large buffer for (say) learning or whatever. What does a >>>>>>large malloc() do to the other program? swap it out? >>>>> >>>>>Never thought of this. This is a real killer, I agree. If done on purpose you >>>>>win every game :-) >>>>> >>>>>>4. A program (ie chessmaster) might poll for input, consuming 1/2 of the cpu >>>>>>even though it is not 'thinking'. >>>>> >>>>>Another true point. Every program needs to poll for input. Asking the >>>>>keyboard / mouse for input are expensive (slow) operations. In Rebel >>>>>I have a counter that makes sure to look for input after 500 evaluations. >>>>>If I decrease the value to say 50 or 10 the NPS of Rebel drops >>>>>tremendously (forgot about the exact slow-down). >>>>> >>>>>Thus the opponent program will ALWAYS slow-down yours and you can >>>>>only guess how much that is. >>>>> >>>>>>There are _too_ many things a program _might_ do. I'll bet not one person >>>>>>gave any thought to a "learning cycle" in crafty, yet it does it in every >>>>>>game. And it steals 2-4 seconds of time from the opponent. In short time >>>>>>controls, that might be important. >>>>>> >>>>>>If I _know_ people are testing like this, I'll bet I can raise Crafty's rating >>>>>>by 100 points minimum. I won't say how, but it shouldn't take too much >>>>>>imagination to figure it out. :) And with that said, why bother testing in a >>>>>>way that is obviously potentially unreliable. For fun, sure. But reporting >>>>>>the results as "A beats B" is not very scientific... A might not actually >>>>>>be able to beat B, he just might have a smarter programmer that takes >>>>>>advantage of a flawed testing methodology... >>>>>> >>>>>>:) >>>>>> >>>>>>Bob >>>>> >>>>>Strong points Bob. Still people are in love with the system as you only >>>>>need one PC and have a lot of fun. If only its results are judged in the >>>>>way it should. And in no way you can compare eng-eng matches on 1 PC with >>>>>eng-eng matches on 2 PC's. >>>>> >>>>>Ed >>>>> >>>>>PS, match score sofar PB_ON vs PB_OFF 17.5 - 8.5 >>>> >>>> >>>>I suspect you are going to get a 100+ point difference. When I first started >>>>running Crafty on ICC, I was experiencing the usual "scanf()" buffering issue >>>>with winboard and couldn't get pondering to work right, although it was fine >>>>in text mode without xboard. When I got it right, crafty's rating instantly >>>>jumped by about 120 rating points, maybe more. When I would turn it off, down >>>>it would go by the same amount. I have, as a result, generally credited >>>>ponder=on with roughly 120 rating points. Be interesting to see how your >>>>results end. You are at about the 120 point level yourself, so far, if my >>>>mental calculations are anywhere near right, winning 2 of every 3 games. 3 of >>>>every 4 gets to +200, so maybe you are closer to 160 than 120... we will see... >>> >>> >>>It's worth pointing out that whatever rating difference he gets is going to be >>>inflated, since he is using the same engine on both machines and therefore >>>getting perfect ponder accuracy. >>> >>>--Peter >> >> >>It won't get 'perfect ponder accuracy' by a long shot. I doubt he will >>predict over 50%. Because one engine is _always_ able to outsearch the other >>by 1 ply on its turn. because both will be doing about the same depth >>normally, but when one does an 8 ply search, it only has info based on a 7 ply >>search by its opponent. When its opponent actually moves it will have 8 plies >>(or whatever) to choose a move. >> >>Pondering accuracy won't be nearly as high as you think... particularly when >>the game starts to turn tactical. > >It won't get perfect ponder accuracy but I believe that it will get more >than 50% if you play the same engine against itself. > >I think the probability to change your mind in the last ply is clearly less than >50% and even more than 60%. >Is there an evidence that I am wrong? > >When the game starts to be tactical you can sometimes get even more accuracy >because the moves of the sides are sometimes forced. > >Uri I find my prediction rate hardly ever is below 50%, whether I play myself, or other computers, or GM players (at longer time controls). At short time controls, GMs make many strange moves and blow this. I haven't looked in a while, but Crafty has played Ferret (for example) recently in a couple of longish games. I'll try to have it play itself to see how the prediction rate goes since it already counts the number of times it is right...
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.