Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Doesn't appear to work for me (full data)

Author: Dave Gomboc

Date: 14:42:47 11/22/02

Go up one level in this thread


On November 22, 2002 at 06:39:47, Gian-Carlo Pascutto wrote:

>On November 21, 2002 at 17:23:15, Dave Gomboc wrote:
>
>>>Time to solution is not a common method in academic computer chess papers. It is
>>>not generic enough, and is too program dependant to be scientifically
>>>acceptable.
>>>
>>>Apart from a few exceptions, I have seen no other publication which discusses
>>>time to solution.
>>
>>When comparing various searches within a single program, nodes to solution is
>>also okay.
>
>I disagree for the reasons below.
>
>>Time to solution allows irregularity because of timing anomalies and
>>more generally because the machines one has access to are not dedicated to your
>>task alone (other people can log on and do stuff).
>
>Easily fixed by getting a dedicated machine.

Are you kidding?  I don't know about you, but I sure can't afford the kind of
CPU power than I can harness at my university!  And I can tell you that if all I
had to use was a single personal computer, my current experimental work would be
_totally_ infeasible.  It is slow even when using dozens of workstations.

>>If the correlation between nodes and time is tightly linear (usually, it is,
>>for a non-tiny search) then using node counts aids reproducibility of results.
>
>Only if NPS is farily constant. That's pretty program dependant, and it
>certainly isn't true for me. Search changes can cause NPS differences.

I'll grant this is something to take into account.  However, the timing
irregularities caused by using machines that have other, varying loads on them
are orders of magnitude larger than these effects.

If the experiment can reasonably be done with limited hardware resources, then I
will agree that reporting cpu time should be done (perhaps also with node
counts, for comparison to other work).  But this does not happen all that often,
usually people are pushing the envelope of what they can feasibly compute with
the resources they have access to.

Dave



This page took 0.14 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.