Author: Komputer Korner
Date: 13:27:49 12/16/98
Go up one level in this thread
On December 16, 1998 at 02:44:38, Christophe Theron wrote: >On December 15, 1998 at 23:41:50, Kevin Mulloy wrote: > >>Gentlemen, >> I am in the process of testing MPC7 in a 200 MMX 80 meg RAM machine against >>MCP7 in a 450PII 256 meg RAM machine. In the 200, the MCP7 engine calculates >>aprox 1,250,000 positions per minute. In the 450, about 2,600,000 positions per >>min (slightly more than double). I have adjusted the times accordingly -- I >>give the 200 2 min per move and the 450 1 min per move. I thought that this >>should be a fairly even match with a slight edge to the 450. After 20 games, >>the 200 leads 15-5?? I was not testing this fine program against itself to try >>to produce a "winning" machine. I just wanted an easy way to handicap one >>machine so that the 200 and the 450 would play fairly even chess when I matched >>different programs against Mcp7. I realize that I have a very small sample here >>-- could someone advise me on the proper set up that I should be looking for to >>even these 2 machines out. Also, If my approach to this problem is OK, how much >>bigger should the sample be to give a good indication of strength? Thank you, >>Kevin Mulloy > >My opinion is that you should not use an average time per move, but use a given >time for the whole game, or classical time controls (A moves in B minutes, then >C moves in D minutes, and so on). > >Depending on the program, average time per move is sometimes completely erratic. >Maybe it is the case with MChess? > >Also, if you want such a match to be fair, you have to disable thinking on >opponents time. The trouble here is that not doing so should have given an >advantage to the fastest computer, so really I cannot explain your result. > >Strange... > > > Christophe The problem of testing the same program against each other on 2 different machines with different speeds (even if you handicap them fairly) is well known. The only way it would be fair is if the program was a stict brute forcer with no extensions and no pruning nor null move nor selective search of any kind. However all programs have some form of pruning so the test is biased. Usually the bias would be for the faster machine if both had the same time control in that the expected win of the faster machine would be more than proportional to the speed advantage. In this case since you handicapped with a longer time control the bias is towards the longer time control. Unfortunately at short time controls, there isn't a perfect correlation between search depth and time control when measuring on different cpus. Hash tables are one example that screws up your search depth handicapping. It makes me think that M-Chess is a larger selective pruner than we have all thought. However your sample is way too small to get any indication of bias. -- Komputer Korner
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.