Author: Albert Silver
Date: 09:56:43 06/25/03
Go up one level in this thread
>>Nope, if the SPEC programs don't produce the correct output, the benchmark score
>>is invalid.
>>
>>This was a nasty thing for the SPEC people to enforce with SPECfp because
>>floating point instructions can give you different results and still be
>>correct...
>>
>>Once in a while companies are caught cheating on the benchmark, e.g., their
>>compilers produce code that works but doesn't match the source, or produce code
>>that's bizarrely well-optimized for the SPEC code but not other, similar code.
>
>Yes, this is akin to engines tuned to positions of certain test suites, but not
>able to maintain the level the results indicate. Ed was once attacked by a
>magazine because he had forgotten that Rebel had been tuned for a specific
>positon, giving the correct result instantly, but had not removed this in the
>release.
>
> Albert
This reminded me of the recent stink on Nvidia's card the GeForce FX 5900 Ultra.
Futuremark, maker of the industry's most common graphics-card benchmark,
3DMark03, accused Nvidia of manipulating test results. Nvidia, of course, denied
cheating. Meanwhile, Nvidia's chief competitor, ATI, admitted that it, too,
could be accused of fudging a little. Futuremark updated the tests and in the
new results scores for Nvidia's GeForce FX 5900 Ultra dropped as much as 22
percent, while scores for ATI's Radeon 9800 Pro 256MB fell only between 2 and 4
percent.
http://news.com.com/2100-1046-1009574.html
Albert
>
>>This reassures me more than it unnerves me, because it shows how much companies
>>are policing each other.
>>
>>-Tom
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.