Author: Aaron Gordon
Date: 09:43:38 11/13/03
Go up one level in this thread
On November 13, 2003 at 12:10:24, Eugene Nalimov wrote: >On November 12, 2003 at 23:22:53, Aaron Gordon wrote: > >>On November 12, 2003 at 13:54:07, Eugene Nalimov wrote: >> >>>On November 12, 2003 at 11:55:20, Gian-Carlo Pascutto wrote: >>> >>>>On November 11, 2003 at 23:42:45, Eugene Nalimov wrote: >>>> >>>>>My point is: it's possible that due to the fact that quad Opteron is NUMA -- >not SMP -- system, for SMP-only program performance on quad Opteron can be >>>>>worse than on *real* quad SMP system, even when for one CPU Opteron >>>>>performance is much better. Itanium was used only as an example of such >>>>>system, I never recommended rewriting any program for it. >>>> >>>>I don't understand how. The NUMA part is RAM. Even worst case on the Opteron >>>>RAM is faster than Xeon SMP. So how could it ever be worse? >>>> >>>>-- >>>>GCP >>> >>>I can think of several reasons why scaling is very bad if all the memory was >>>allocated at one CPU: >>> >>>(1) Memory *bandwidth*. All the memory requests go to exactly that CPU, so all >>>CPUs have to use exactly one (or two) channels to memory. On Xeons *worst case* >>>memory bandwidth is higher. >>> >>>(2) CPU-to-CPU *bandwidth* -- memory transfer speed is limited by the fact that >>>*one* CPU has to process memory requests for for *all* CPUs. Also notice that >>>for "normal" topology >>> >>> 0----1 >>> | | >>> | | >>> 2----3 >>> >>>CPU#3 has to go through either CPU#1 or CPU#2 to reach memory of CPU#0. >>> >>>(3) MOESI vs. MESI synchronisation protocols -- I was told that on MOESI (used >>>by AMD) traffic due to shared *modified* cache lines is much higher than on MESI >>>(used by Intel). If it is really so (I didn't investigated myself) it probably >>>can explain why on 32-bit Athlons Crafty prior to 19.5 scaled worse than on >>>Pentium 4. >>> >>>In any case here are results of Crafty 19.4 scaling on 2 different Opteron >>>systems, and on Itanium2 system (measured before Crafty became NUMA-aware, and >>>we decreased amount of shared modifiable data): >>> >>>Opteron system I: >>>2 CPUs: 1.57x >>>3 CPUs: 1.99x >>>4 CPUs: 1.98x >>> >>>Opteron system II: >>>2 CPUs: 1.61x >>>3 CPUs: 2.13x >>>4 CPUs: 2.35x >>> >>>Itanium2 system: >>>2 CPUs: 1.84x >>>3 CPUs: 2.63x >>>4 CPUs: 3.22x >>> >>>Crafty 19.5 scales much better. On Opteron system II it reaches 3.8x on 4P. >>> >>>Thanks, >>>Eugene >> >>So, are you saying it needs special NUMA code to get 'full' bandwidth and that >>it defaults to a single memory channel? Running Windows 2k, XP, and other SMP >>operating systems the Opteron *always* gets the full memory bandwidth across all >>of its cpus. Hardware test pages ran all kinds of reviews/tests and every single >>one showed a dual/quad pulls a ridiculous amount of bandwidth. Remove the chips >>and it goes back down accordingly. > >Can you please point me to the test that allocates all the memory on one CPU, >and then *all the CPUs* read and write to *that* memory? I am not talking about >SPECrate type of test where you just run several independent (or almost >independent) processes simultaneously. > >Thanks, >Eugene Fire up Sciencemark 2.0 (www.sciencemark.org) and/or Sisoft's Sandra (http://www.sisoftware.co.uk/index.html?dir=dload&location=sware_dl_x86&langx=en&a=).
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.