Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Everything you know is wrong

Author: Vincent Diepeveen

Date: 04:47:01 12/18/02

Go up one level in this thread


On December 18, 2002 at 05:49:43, Sune Fischer wrote:

>On December 18, 2002 at 03:08:03, Bruce Moreland wrote:
>
>>You have version A which gets 10 seconds per position to think on some
>>hypothetical hardware, and finds 65 solutions.  You have version B, which gets
>>17 seconds on the same hardware, and finds 71 solutions.
>>
>>Which is better, A or B?
>>
>>You conclude B.
>
>There can be no conclusion, because although version B solves more position it
>also uses more nodes.
>
>>I disagree with your conclusion, for obvious reasons.  *If* Ernst does it the
>>same way, I disagree with Ernst.  And *if* Bob does it that way, I disagree with
>>Bob.
>
>I don't believe he concluded that, I asked Omid that question last time the
>discussion was running here, and he said there was no conclusion to be drawn in
>that case, which of course is correct.

I agree. You can't draw conclusions based upon that.

>>Which is my point.  If this paper can be juried and still published with this
>>flaw in it, and if foremost experts in the field can fail here to refute this
>>methodology, our field is stuck and needs a good kick.
>>
>>You cannot conclude that algorithm B is superior to algorithm A, if you give
>>algorithm B more time to operate.
>>
>>The proper conclusion is that you don't know if B is superior to A.
>
>The problem as I see it, is that he has split the process in two.

Right he has wrongly split it in order to avoid even the worst proofreader
from smelling the truth.

>First he designs an algorithm to make a smaller tree and then he verifies that
>it's also better (solving more positions).

In neither of the 2 tests the algorithm is superb. So he could not
draw the conclusion that verification search is better. *no way*.

>Those are very though demands, you _can_ get an overall improvement by say,
>searching 5% more nodes but in return get far less instability in your search.

He doesn't hash whether he has done a verification, so his implementation
for sure is buggy. I wonder why proofreaders didn't see that.

>Such an improvement wouldn't never be discovered by this method, because more
>nodes is bad _by definition_.

Not always. time to solution counts. Nothing else counts.
Of course there is other conditions that must be satisfied too
like testing at the same machine.

If i do not nullmove in DIEP at 1 ply depthleft, then diep needs less
nodes to complete a search, but more time. Number of nodes is trivially
bad way to measure things when we talk about nullmove and such methods
that all put stuff into the hashtable.

Nothing as cheap as a hashtable transposition. Just like 500 clocks or
so. There is plenty of other algorithmic changes to figure out which
reduce number of nodes and increase time to search. Good other example of
what decreased my node count *considerably* but is just too expensive
too do is ordering at every position in the main search all moves based
upon evaluation of the position after it (of course also using a SEE
with it in combination). So ordering it by evaluation.

Even if i do not do that at depthleft == 1 ply, then it is slowing me
down a factor 2 or so nearly but it reduces node count with around
30% or so?

>Like others, I agree that a more "objective" form of measure should be done, ie.
>nodes to solution / time to solution.

time to get to the same search depth and if you fear you go prune
too much then also how many positions you solve within a certain
time.

Nodes to solution is not acceptible at all.

>Personally I prefer nodes to solution, because of the reproducibility. I don't

In a parallel search nodes to solution are not reproducable. Also the
best parallel algorithm is keeping all but 1 cpu busy (for example
dead locked) and letting 1 processor search. A 100% speedup i get then
if i just look to nodes to solution. don't you agree that this
leads to conclude that we have created the ideal algorithm then?

>like having to do averages when it can be avoided. Also if Omid is running on
>some grand network machine with different loads from minute to minute, it is
>going to be meaningless.

that is not the case here clearly. a scientist has to take care his
tests are done accurate. So you publish both. simply the entire output
of the program.

If the entire output of genesis would be on his homepage, we still can
draw a diagram whether R=3 solved within the same time more solutions
than R=2.

So publishing the logfiles of ones program is something you need to do
to IMHO. I sure will.

>However if he plans to do parallel algorithms, then I think he really neesd a
>constant dedicated resource and time to solution measurements.

You cannot compare testing at a home computer with parallel testing at
a supercomputer. Like Bob explained already it is very hard to get system
time. For me it's easier now to perform tests, but i also do not know
at which partition DIEP gets scheduled. there is 7 partitions.

I am at the mercy of the OS where the program is scheduled. If i ask for
16 processor test of DIEP then it gets scheduled where the resource
is free earliest. If that's at the P2 partition which has
32 processors then communication there is a lot faster
than at the P7 partitions which has 512 processors. Also it is never
the case that the P7 partition is idle.

This machine is too expensive to get exclusively used by me for just
a few tests. No government can afford that, for sure not the Dutch
Government.

If you want to know what a test costs that's not so hard.

For example for a short 16 processor test of less than 3 minutes that
was in total 1800 seconds cpu time used actually. So that was calculated
at 0.5 cpu hour.

If i do a test at 256 processors for 1 day. that's

256 processors x 24 hours a day x 30 euro a processor hour = 184320 euro.

DIEP at world championship 2003:
  7 days (they keep entire partition free for me) times
  24 hours
  512 processors
= 86016 processor hour
  at 30 euro a processor hour = 2580480

Slowly realize the problem?

Someone shouting that it ain't 30 euro a processor hour?
Ok then do next math: divide half a billion dollar (cost of
machine) by the number of processor hours available within a
year or 3. Don't forget to also include power, support from SGI,
and many salaries of doctors running the thing and expenses of
professors and clever scientists taking decisions what happens with
the machine.

Let's do it for 3 years:
0.5 billion euro / 1024 processors x 350 days (15 days maintenance) x 24 hours
times 50% effective usage of the machine at any moment of the day =

500 000 000 / 50%.1024.350.24.3 = 38.75 euro for each processor hour.

These numbers are fictive and are not true for the TERAS machine.
I invented them. Be clear. It is just an example to give insight in
the costs of those machines for each cpu hour.

It is clearly you can't demand from supercomputer runs that you always
have the same bandwidth and latency for every test. Despite that a
researcher should of course try to do his best. Important is to not
invent numbers yourself. Or to lose your precious runs.

Note that outputs from any run i ever did are saved in special
logfiles and error files. No i am not deleting them. Instead i keep
track of them and when DIEP's working that well that i feel i must
do a few big tests (instead of the 3 minute tests i keep now at at most
16 cpu's) then all those outputs will be freely available for download,
upload and spreading around the world as long as they are unmodified.

I wonder always where the outputs of the previous parallel researchers
are... ...i know they do not get thrown away. The back up facilities
at such supercomputers are great!

Best regards,
Vincent

>>It is possible that your algorithm is superior.  Further testing may show it.
>>The data as reported do not support your conclusion.
>>
>>If you were trying to prove that this is superior to null move R=2, your data
>>can support that conclusion, since you found more solutions with fewer nodes.
>>That is a very interesting conclusion.
>>
>>But your data do not show that your algorithm is superior to null move R=3,
>>since you found more solutions with more nodes.  An inferior algorithm can do
>>that.  You could have tested the same algorithm twice and proven that it is
>>superior to itself.  Something is wrong here.
>>
>>bruce



This page took 0.04 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.