Author: Ed Schröder
Date: 23:35:31 06/27/99
Go up one level in this thread
>Posted by Robert Hyatt on June 27, 1999 at 21:15:23: > >>>I can have Brett's opinion,too, and the evidence is abundant. In the last >>>several weeks results of games between H7.32 and F5.32 on separate computers >>>have consistently shown the two programs to be close or have shown F5.32 >>>to be better at blitz games; at the same time engine v engine results >>>have been posted over and over showing H7.32 to be almost 200 elo points >>>better than F5.32 at various time controls, including blitz. This difference >>>cannot be explained, except that engine v engine is a different animal than >>>computer v computer. >>> >>>eric >> >>My results on one computer have not shown this. I see this type of stuff >all the >>time when some people post results by it on one computer or two. How >knowns how >>they test or if they are showing the real and accurate results. I can only >say I >>have seen nothing to say that you can not get accurate results with the >>chessbase interface playing it on one computer. If the data was scewed in >a big >>way I would see it and it would not match up with my two computer results or >>other reliable two computer results. > >I can only offer two data points. In Crafty, _my_ timing allocation code >assumes that there will actually be more time to use than crafty has at the >point it has to make the decision. Because I _know_ that I will correctly >predict some moves here and there, and save that time. And do I want to wait >until I save it before I use it, or should I use it in the part of the game >where it is important, rather than taking 10 minutes per move in a simple >endgame? > >The other data point was an old Rebel. Ed did all of his time setting while >"permanent brain" was being used. When he turned it off in the NPS match he >saw bad time allocation too. > >I don't know about others, but that is significant enough. > >Another point is that some programs (again, mine is an example) depends on >fairly fast hardware, because of some of the search decisions I have made in >the design process. Cut the speed by 1/2 and it might hurt me more than >another program. Ask Thorsten about the Crafty vs CSTal match a couple of >years ago. He was using p5/90's and Crafty was getting killed. When he >went to something comparable to the P6/200, Crafty won way more than it lost. > >Your "ponder=off" type matches effectively cut the processor speed by 1/2, >when you think about it. And the results can definitely be affected.. I entirely agree with you on both points (a) permanent and (b) time control which are closely connected by each other. The main problem of these engine-engine matches on one Pc is that because the lack of the permanent brain the time control rules of the program in question is not in harmony anymore which may result in very bad play. Playing without permanent brain requires *another* set of rules for time control. Programs who play engine-engine matches on one Pc (a) should realize it is playing such a match and then (b) use a totally different (and well tuned) time control. The big question is if these engines are aware of this. If engine_X knows and engine_Y does not know then engine_X has a big advantage (50-100 elo?!) and results are meaningless in the sense that results can completely different if you run them on 2 Pc's (which remains the only accurate way BTW). To interpret engine-engine matches on 1 Pc in the right way you should know first about all engines if the above knowledge is implemented in engines. The only one who can tell you is of course the programmer. I know engine-engine matches on one Pc is becoming very popular because it is so convenient but if an engine is lacking a new set of rules for time control then results are suspect of not being accurate at all and can be very misleading. As Bob already pointed out above I noticed this phenomenon first playing the NPS game against Crafty. Because permanent was set to off Rebel's time control completely messed up and would have caused a big problem if Bob would not have allowed me to manually change the time control so now and then (trying to represent real 40/2:00) which was very sportsmanlike of Bob. I am 100% sure that without this Rebel would have lost this game without chance because of time control problems. Bottom line: ask programmers who allow engine-engine matches if time control is "ok" in engine matches on 1 Pc. Ed Schroder
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.