Author: Dave Gomboc
Date: 14:42:47 11/22/02
Go up one level in this thread
On November 22, 2002 at 06:39:47, Gian-Carlo Pascutto wrote: >On November 21, 2002 at 17:23:15, Dave Gomboc wrote: > >>>Time to solution is not a common method in academic computer chess papers. It is >>>not generic enough, and is too program dependant to be scientifically >>>acceptable. >>> >>>Apart from a few exceptions, I have seen no other publication which discusses >>>time to solution. >> >>When comparing various searches within a single program, nodes to solution is >>also okay. > >I disagree for the reasons below. > >>Time to solution allows irregularity because of timing anomalies and >>more generally because the machines one has access to are not dedicated to your >>task alone (other people can log on and do stuff). > >Easily fixed by getting a dedicated machine. Are you kidding? I don't know about you, but I sure can't afford the kind of CPU power than I can harness at my university! And I can tell you that if all I had to use was a single personal computer, my current experimental work would be _totally_ infeasible. It is slow even when using dozens of workstations. >>If the correlation between nodes and time is tightly linear (usually, it is, >>for a non-tiny search) then using node counts aids reproducibility of results. > >Only if NPS is farily constant. That's pretty program dependant, and it >certainly isn't true for me. Search changes can cause NPS differences. I'll grant this is something to take into account. However, the timing irregularities caused by using machines that have other, varying loads on them are orders of magnitude larger than these effects. If the experiment can reasonably be done with limited hardware resources, then I will agree that reporting cpu time should be done (perhaps also with node counts, for comparison to other work). But this does not happen all that often, usually people are pushing the envelope of what they can feasibly compute with the resources they have access to. Dave
This page took 0.01 seconds to execute
Last modified: Thu, 07 Jul 11 08:48:38 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.