Author: Albert Silver
Date: 09:44:46 06/25/03
Go up one level in this thread
On June 24, 2003 at 16:40:13, Tom Kerrigan wrote:
>On June 24, 2003 at 16:10:13, Daniel Clausen wrote:
>
>>On June 24, 2003 at 16:01:34, Dann Corbit wrote:
>>
>>[snip]
>>
>>>If they chose GCC, it is probably because GCC produces a better binary than
>>>anything else at their disposal. And if that is the case, then we are certainly
>>>seeing the true measure of performance. After all, anyone else who creates
>>>software tools will have to use that compiler or an inferior one also.
>>
>>OTOH, building a compiler around SPEC can also be misleading. :)
>>
>>I guess both sides shouldn't take these benchmarks too serious anyway.. I
>>definitely don't like the way Apple produced the benchmarks, but I'm sure
>>they're not the only company who 'tweak' the results a little to their liking..
>>there are many ways to do this. SPEC(Int|FP) (and others benchmarks) surely show
>>the general direction, but since the companies involved have so much
>>self-interest in the results it's hardly very scientific anyway.
>>
>>
>>bool heavyCalculation(lots_of_input_data)
>>{
>>#ifdef ITS_LIKELY_I_AM_USED_AS_A_BENCHMARK
>> return true; // Change to 'false' on Tuesdays
>>#else
>> return reallyDoTheHeavyCalculation(lots_of_input_data);
>>#endif
>>}
>>
>>and compile it with 'gcc -fomit-instructions' :p
>>
>>Sargon
>
>Nope, if the SPEC programs don't produce the correct output, the benchmark score
>is invalid.
>
>This was a nasty thing for the SPEC people to enforce with SPECfp because
>floating point instructions can give you different results and still be
>correct...
>
>Once in a while companies are caught cheating on the benchmark, e.g., their
>compilers produce code that works but doesn't match the source, or produce code
>that's bizarrely well-optimized for the SPEC code but not other, similar code.
Yes, this is akin to engines tuned to positions of certain test suites, but not
able to maintain the level the results indicate. Ed was once attacked by a
magazine because he had forgotten that Rebel had been tuned for a specific
positon, giving the correct result instantly, but had not removed this in the
release.
Albert
>This reassures me more than it unnerves me, because it shows how much companies
>are policing each other.
>
>-Tom
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.