Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Verified Null-Move Pruning, ICGA 25(3)

Author: Omid David Tabibi

Date: 04:00:01 11/22/02

Go up one level in this thread


On November 22, 2002 at 06:49:45, Gian-Carlo Pascutto wrote:

>On November 22, 2002 at 06:42:39, Omid David Tabibi wrote:
>
>>Comparing search times has the following problems:
>>
>>1. It is not reproducible:
>>
>>    - If you manage to speed up your program (without algorithmic change),
>>      the results will be different.
>
>
>Huh? You should never make such changes when you are already testing
>your algorithm!

But if someone asks me to reproduce my results a year later?

>
>>    - If you move to another platform, the results will be different.
>
>Howso? The absolute times will be different yes. The ratios, not.
>

But again, these are not the very same results.


>>    - Even on the same very hardware with the same very program, you will
>>      not get the exact results of the previous test.
>
>Not exact, but close enough. Within 1% in my experience.
>

"Not exact".


>>2. It is not generic:
>>
>>    - dependant on your program's structure
>>
>>    - dependant on your program's speed
>
>The same is true for other methods.
>
>>    - dependant on your hardware
>
>See above.
>
>>3. It is not accurate:
>>
>>    I have conducted tests on thousands of positions during the past several
>>    months. Sometimes the computer was busier, sometimes it crashed and I had
>>    to restart it (Windows!), etc.
>
>Use a dedicated computer. If Windows is causing problems, use a Unix box.

Haven't your university's UNIX servers run through any problem in the past six
months?

>

You see, fixed time poses many problems that have to bridged. But fixed depth
comparison, will result in the very same results, no matter what the hardware
is. That is how a scientific research should be conducted (as reproducible as
possible).

Had I posted the time comparisons, you would have asked:
- what is your hardware?
- what is your program's NPS?
- is it a stable NPS?
- hasn't the computer crashed while conducting the tests?
- has it been a dedicated computer?
etc...

But now that I have presented my results in form of node counts, the above
questions are irrelevant.


>--
>GCP



This page took 0.08 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.