Author: Tom Kerrigan
Date: 13:40:13 06/24/03
Go up one level in this thread
On June 24, 2003 at 16:10:13, Daniel Clausen wrote:
>On June 24, 2003 at 16:01:34, Dann Corbit wrote:
>
>[snip]
>
>>If they chose GCC, it is probably because GCC produces a better binary than
>>anything else at their disposal. And if that is the case, then we are certainly
>>seeing the true measure of performance. After all, anyone else who creates
>>software tools will have to use that compiler or an inferior one also.
>
>OTOH, building a compiler around SPEC can also be misleading. :)
>
>I guess both sides shouldn't take these benchmarks too serious anyway.. I
>definitely don't like the way Apple produced the benchmarks, but I'm sure
>they're not the only company who 'tweak' the results a little to their liking..
>there are many ways to do this. SPEC(Int|FP) (and others benchmarks) surely show
>the general direction, but since the companies involved have so much
>self-interest in the results it's hardly very scientific anyway.
>
>
>bool heavyCalculation(lots_of_input_data)
>{
>#ifdef ITS_LIKELY_I_AM_USED_AS_A_BENCHMARK
> return true; // Change to 'false' on Tuesdays
>#else
> return reallyDoTheHeavyCalculation(lots_of_input_data);
>#endif
>}
>
>and compile it with 'gcc -fomit-instructions' :p
>
>Sargon
Nope, if the SPEC programs don't produce the correct output, the benchmark score
is invalid.
This was a nasty thing for the SPEC people to enforce with SPECfp because
floating point instructions can give you different results and still be
correct...
Once in a while companies are caught cheating on the benchmark, e.g., their
compilers produce code that works but doesn't match the source, or produce code
that's bizarrely well-optimized for the SPEC code but not other, similar code.
This reassures me more than it unnerves me, because it shows how much companies
are policing each other.
-Tom
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.