Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Verified Null-Move Pruning, ICGA 25(3)

Author: Omid David Tabibi

Date: 04:18:48 11/22/02

Go up one level in this thread


On November 22, 2002 at 07:06:21, Gian-Carlo Pascutto wrote:

>On November 22, 2002 at 07:00:01, Omid David Tabibi wrote:
>
>>>Huh? You should never make such changes when you are already testing
>>>your algorithm!
>>
>>But if someone asks me to reproduce my results a year later?
>
>Save the version you used. I don't think this is even needed, see
>Roberts PhD :)
>
>>>Howso? The absolute times will be different yes. The ratios, not.
>>>
>>
>>But again, these are not the very same results.
>
>So what? Does that alter the conclusion of your paper? It better
>had not!
>
>>>>    - Even on the same very hardware with the same very program, you will
>>>>      not get the exact results of the previous test.
>>>
>>>Not exact, but close enough. Within 1% in my experience.
>>>
>>
>>"Not exact".
>
>Have you ever learned something about experimental error? This is one.
>Easily accounted for with _really_ basic statistics.
>
>>>Use a dedicated computer. If Windows is causing problems, use a Unix box.
>>
>>Haven't your university's UNIX servers run through any problem in the past six
>>months?
>
>So? Rerun the test in question. Besides, for testing I have a dedicated
>box that I run myself. Linux is free, a leftover computer isn't exactly
>hard to find either.
>
>>You see, fixed time poses many problems that have to bridged. But fixed depth
>>comparison, will result in the very same results, no matter what the hardware
>>is. That is how a scientific research should be conducted (as reproducible as
>>possible).
>
>You can do this perfectly with fixed time as well, your results will be
>perfectly within the statistical error.
>
>>Had I posted the time comparisons, you would have asked:
>>- what is your hardware?
>>- what is your program's NPS?
>>- is it a stable NPS?
>>- hasn't the computer crashed while conducting the tests?
>>- has it been a dedicated computer?
>>etc...
>
>You think so? Please take a look at the results I posted when
>I tested your algorithm. I never mentioned hardware, NPS, what
>computer or OS I used because they are irrelevant.
>

Because we were discussing the performance on your specific program. But if we
wanted to draw a wider conclusion regarding the algorithm, we had to know all
those details.
Besides, you might agree that the results you posted with fixed time, while
being practical for testing your program, are not general enough to be
publishable.


>>But now that I have presented my results in form of node counts, the above
>>questions are irrelevant.
>
>No, not at all.
>
>--
>GCP



This page took 0 seconds to execute

Last modified: Thu, 15 Apr 21 08:11:13 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.