Author: James T. Walker
Date: 08:58:32 07/27/00
Go up one level in this thread
On July 26, 2000 at 14:17:30, Gordon Rattray wrote: >I've been attempting to benchmark two different machines. So, I ran Junior 6a on >a given position and saved the analysis. I repeated a check test *on the same >machine* to see if the output would be any different, but I got the same >analysis (as expected!). So I concluded that I had obtained some valid results >for that particular machine. > >However, when I tried the same using Fritz 6a, it was difficult to get the same >results twice! In fact, for one test position, during the first run it found >the key move after about 2 minutes. When I ran it again, it required 6 minutes! > >Ok, so something is changing between test runs, but what? Here's some facts >that may provide some clues... > >- the machine has a total of 64Mb RAM. 32Mb are being used for hash. There is >no disk access during the test runs (I definitely watched for this!) > >- endgame table bases get accessed for some of my test positions > >- when I first start Fritz, I allow it an initial "dummy analysis run", just to >get its hash table space setup. I discard this test run. > >- I'm using Windows 98 > >- before each test run, I perform a "clear hash tables" on the engine > >- during each test run, no other applications are being run on the PC, just >Fritz. I'm also fairly certain that Windows doesn't start any background tasks >such as it's scheduled system maintenance tasks. > >- when the test has run for long enough, I clip the analysis and only then run a >word processor. I quit the word processor before performing the next run. I >then run another "dummny analysis run" to try to allow Fritz to get it's memory >space sorted out again (i.e. get any memory paging done), before starting the >next test run. > > >I don't expect results to be repeatable to the exact second every single time. >Maybe some minor factors may throw things out slightly - fair enough. But I >can't understand why I'm seeing such significant differences?? 2 minutes for >one run, 6 minutes for the next! > >Has anyone else seen this? Or do you know what is responsible for changing the >behaviour so drastically? I suspect my testing procedure doesn't support >consistent runs, and despite trying to be careful, I'm still getting caught >out?! Why is Junior not being affected similarly? > >(Finally, I apologise for not appending any "clipped analysis" here as I don't >have the results on this machine. I also don't think that it will help identify >the cause, but if this assuption is wrong, I'll forward the analysis a.s.a.p). > >Thanks in advance for any suggestions > >Gordon Hello Gordon, How long did you let Fritz run before ending the analysis? I let Fritz run for 8 minutes on the position you provided and the analysis was identical except for the second run being slightly faster in reaching 20/36 plies. It was 2:20 in the first run and 2:18 in the second. Also at the 21/40 plies mark it was 6:40 for the first run and 6:32 for the second. I have no idea what could be causing your differences unless you are running a quick analysis like 5-10 seconds. All other statistics were identical. (Kilo nodes & tablebases hits) Jim Walker
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.