Author: Robert Hyatt
Date: 18:47:01 09/03/02
Go up one level in this thread
On September 03, 2002 at 20:29:19, Rolf Tueschen wrote:
>On September 03, 2002 at 17:30:35, Robert Hyatt wrote:
>
>>As is usually the case, someone that helped me with this had sent an email
>>while I was responding to the other posts. And when I read the second
>>paragraph, it all "came back".
>>
>>Here is the issue:
>>
>>I started with a 16 cpu game log file. Note that this was from a real
>>game. And in it I would find output just like Crafty's... Here is the
>>idea:
>>
>> depth time eval PV
>>
>>followed by a summary.
>>
>>The problem is that the node count in the summary has nothing to do with the
>>PV when it was displayed. The program _could_ have stopped the search as soon
>>as the PV was displayed, or it could have stopped the search minutes later.
>>As a result, I had no real node counts for the 16 cpu test that could be
>>compared to anything else since there was no way to know when the 16 cpu
>>test completed.
>>
>>We chose to do the following:
>>
>>1. run the positions thru a one processor search, and since there was no
>>parallel searching going on, we could display an _exact_ node count for the
>>one-processor test, as it would have been had the search stopped immediately
>>after producing the critical PV move at the final depth. That value _is_ a
>>raw data point.
>>
>>2. We then ran the positions thru the 2-processor search, taking the time
>>for the same PV as the time. All the times are pure raw data, exactly. But
>>we couldn't get a good node count. What we chose to do was to use an internal
>>performance monitor we had built in, that very precisely told us how much cpu
>>time had been spent playing chess by each processor. From these times, we
>>computed speedups for 2 processors, 4, 8 and 16 (we didn't run the 16 cpu test
>>again, we just used the raw log from the mchess pro game...
>>
>>3. We now had a set of speedups for each test. Which we plugged into the
>>article. And again, it is important to note that for this data, the raw
>>speedup was computed by dividing the times as you would expect.
>>
>>For the node counts, which was impossible for us to obtain from any but the
>>one processor test, we simply extrapolated them based on the cpu utilization
>>of all the processors. Some simple testing by searching to a fixed depth on
>>one processor and then 16 processors shows that our "extrapolation" was "right
>>on"... and we used those node counts.
>>
>>4. Clearly, the node counts are therefore produced from the raw 1-cpu data,
>>multiplied by the percent of cpu utilization for the 2,4,8 and 16 cpu test
>>cases. So they should correlate 100%.
>>
>>The only thing that my (nameless) partner said was that he could not remember
>>if we did the same thing to produce the times since it would have been easier
>>than trying to extract them from the logs later to produce the table for times.
>>He "thought" that the times were added after a request from a referee, so that
>>is possible.
>>
>>So, perhaps the data has some questionable aspects to it. The only part that
>>I am _certain_ is "raw data" is the individual speedup values, because that is
>>what we were looking at specifically. I had not remembered the node count
>>problem until this email came in and then I remembered a case where Vincent
>>was trying to prove something about crafty and got node counts suggesting that
>>it should have gotten a > 2.0 speedup. I had pointed out that the way I do
>>nodes, it is impossible to produce them anywhere except when all processors are
>>idle, if you want an accurate number. I _should_ have remembered that we had
>>the same problem back then. I am therefore afraid that the times might have
>>been computed in the same way since it would have been quite natural to do
>>so...
>>
>>I don't think this changes one iota about what is going on, of course. as
>>given a speedup, and total time used by Crafty, I can certainly compute a
>>node count that will be _very_ close to the real one. Which I supposed I should
>>add so that Vincent can have his "every time the PV changes give me nodes"
>>type of value.
>>
>>Keep in mind that this was an email from someone that worked on this with me
>>back then. His memory was somewhat better because he actually wrote the code
>>to solve the problem. But again, he was _very_ vague in remembering everything.
>>It took a phone call for us to discuss this to get as far as I did above. I
>>might remember more as time goes on.
>>
>>But the bottom line is "trust the speedup numbers explicitly". And if you
>>trust them, the others can be directly derived from them. For 16 cpus, Cray
>>Blitz generally searched 100% of the time on each cpu. If it produced a speedup
>>of 16, then each cpu searched 1/16th the total nodes searched by one processor.
>>If it produced a speedup of 8, then each cpu searched 1/8 of the nodes searched
>>by one processor, which is 2x the total nodes, aka search overhead.
>>
>>Sorry for the confusion. Stuff done 10 years ago is difficult enough.
>>Remembering the "log eater" was harder since I didn't write all of it...
>>
>>Bob
>
>Bob,
>would you be so kind to explain from your standpoint (and also in the light of
>the adding here) how Vincent's conclusions could be understood?
What he finally said was that either the nodes or times (I don't remember
which) are perfect multiples of the speedup numbers. IE a few of the
results where CB said 6.1 speedup showed that either the nodes or the
times (one of them or both for all I know) was exactly 6.1X the one cpu
result, which would be unlikely to happen more than a few times. And he
was correct.
Unfortunately, he never asked about that at all, in private, otherwise I
could probably have come up with the right answer. But he chose to make
a public fuss about it so that his sponsor will not look at the Cray Blitz
speedup as "real"... so that his speedup (which is quite obviously well below
11.1 on 16 cpus) will not look so bad to them.
I believe that was _the_ reason. To make himself look good to the group that
has promised to give him machine time... If they see this mess, that might
change, of course.
>I mean, you made
>a highly reasonable impression with your answers except the ugly 'medication'
>chapter. (Please note that I do not support Vincent's vocabulary either when he
>spoke of lies and mass fraud, as if it would have been proven yet; to Vincent:
>you might find strange things but the proof of intentional fraud and or lying is
>certainly something else!)
>Would you admit, that from what Vncent could read in your article ('THe Page'),
>he might have forcedly come to his (wrong) conclusions?
>
>
The normal approach would be to notice something that seems inconsistent,
and then approach the author. Heaven knows he has approached me to borrow
my machine for tournaments. He has approached me asking for help in his
parallel search. He has approached me asking for help in improving his NUMA
implementation. So why he didn't approach me here is only answerable with
conjecture.
>Why am I writing this?
>
>Bob, if you want it or not, for the young here you are a 'fixed star', also
>because no one else is showing up from the past. I remember how it became clear,
>and you wrote it, that Vincent could impossibly reach a sound performance with
>the super computer in the actual event. So, his frustration seems explainable.
I explained that to him, _carefully_. And told him _nobody_ would expect a
parallel search on an inferior NUMA machine to come anywhere near a parallel
search on a real cross-bar memory system on a sixty million dollar super-
computer. It would be similar to Volkswagen loaning him a car and then being
upset that he got rolled up by an F-1 or diablo... There are expectations,
and then there are reasonable expectations. Vincent hasn't learned the
difference between them, and I doubt he will for a long time.
IE I never expected Crafty's parallel search to be as good as Cray Blitz's.
For several reasons. But I'm not going to badmouth what we did for CB just to
make my current search look better...
>Also, please note, that Vincent isn't a scientist yet. He's a good chessplayer,
>but by far not IM or GM. His good chess surely is a big motivation for him, but
>I fear that his 'science' is not based on a sound education yet. As far as I am
>informed. He might be an excellent programmer, but this still doesn't help to be
>a sound scientist. I can say that because Vincent made a lot of unneccessary
>verbal attacks, which are completely unknown in real science.
Correct. His definition of "proof" is also sorely lacking...
>
>But why I make this appeal to you? Simply because I at your age and with your
>status and your successes, I would be happy if I had a young man writing and
>searching the way Vincent is doing it. I wouldn't feel insulted at all. And I
>saw that you also stayed cool most of the time. Sure, it'salso not really nice
>that Vincent gets on your throut after he cooperated with you to his own
>benefit. But then I remember how the also young T. Kerrigan communicated with
>you. Also in a very impolite manner. And you did _never_ complain. So,please
>let's give Vincent the same chance. I for one would be happy if my old articles,
>if I were you or some collegue of the past, would be checked so carefully no
>matter with what motivation. Even you had the chance now to remember some
>long-time forgotten stuff.
I'm not particularly insulted by him. Because I _really_ don't take him
seriously any longer. After the >2 speedup nonsense, which he has not finally
dropped after fixing enough bugs, I decided that any sort of "scientific"
discussion with him would be impossible. Good chess player? Undoubtedly.
Good programmer? Debatable. Good computer scientist? Not by a mile.
>
>I think that even if people make grave mistakes, we should stay cool and explain
>the basics, in special if we are teachers!
I am always willing to try this. When he first brought this up last week,
without any specifics other than "your 11.1 is provably impossible" I told
him what I did and how I did it. I didn't think about the node problem
because he didn't mention that at all, and I was focused on the speedup
issue which seemed to be his "personal demon".
>
>So, please you should reconsider the first verdict about Vincent. He's still
>worth your helping hand. The details might be better discussed in private email
>between you two.
Sorry, but not a chance. The old "fool me once, shame on you, fool me twice,
shame on me" applies here. I'm not going beyond twice...
>
>To Vincent:
>
>Your original article is one of the best here. Also if your conclusions are not
>soun d and the language is too impolite. But I can almost feel the temperament
>and the fever that is burning in you.
>
>Please try to get a few lessons in science, I mean the basics, logic and stuff
>like that. The chain is very feable: your undoubtable observations, the
>impossibility to reproduce something, your reading of the article, your
>interpretation then, very limited as many here have told you, and then the
>conclusions with intentional fraud and conscient lies. This is not what is
>happening very often in science! But errors, sloppiness, confusion, zeitnot,
>influences from outside (helping hands) often happen, that's true. But the proof
>that Bob commited all those "crimes" is not there! The main mistake in your
>article is the short time you had to make such objections and that you surely
>didn't show the article to good scientists before! Next time you should do that.
>I mean scientists not students. People with experience. Because if you don't do
>that, then you might fall into trivial traps. Here, I guess but don't know for
>sure, the question of the data-eater. Only one point. Of course there could be
>mistakes in Bob's article, but even that wouldn't necessarily mean that he
>cheated in his dissertation. BTW did you check this one too?
>
>And still I think that your article is very important. It gives us a good
>description of the mess you are in since you began your work on the super
>computer for the Wch. If you can overcome the actual difficulties you are a good
>candidate for future titles! Please try to preserve your "fever", but please
>reduce the heavy artillery words in your vocabulary. In science it's enough if
>you present your data and make hypotheses. The rest, possible ugly conclusions
>is a natural consequence then. No need to push your case like attorneys in
>American courts.
>
>
>GENS UNA SUMUS
>
>Rolf Tueschen
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.