Computer Chess Club Archives


Search

Terms

Messages

Subject: Re: Automatic Eval Tuning

Author: Graham Laight

Date: 14:09:50 07/05/01

Go up one level in this thread


IMO, the thrust of Vincent's argument is that tuning cannot work because
parameters have to be tuned over too few a sample of games.

I would argue that it must be possible - because the evolution of living
creatures is basically a giant automatic tuning operation. This involves huge
numbers of interrelated parameters - and a relatively tiny number of "games" to
sort them out.

What artificial life experiments have shown is that when "creatures" are
evolving in the computer, they don't appear to change much - but in reality,
under the surface, they are "preparing themselves" for change. Then, suddenly,
the new adaptation appears to have happened very quickly.

It is wrong to say it can't be done - we just haven't tried the successful
method yet.

-g

On July 05, 2001 at 07:44:15, Vincent Diepeveen wrote:

>On July 04, 2001 at 09:12:35, JW de Kort wrote:
>
>>On June 29, 2001 at 14:05:06, Jon Dart wrote:
>>
>>>On June 29, 2001 at 11:14:34, Artem Pyatakov wrote:
>>>
>>>>I am curious, have people here experimented or extensively used Eval Function
>>>>tuning based on GM games for example?
>>>>
>>>>If so, is it effective to any extent?
>>>
>>>There is a program called KnightCap that implemented eval
>>>learning and it worked quite well. Source is available.
>>>See http://syseng.anu.edu.au/lsg/knightcap.html.
>>>
>>>--Jon
>>
>>Hi i tried to understand what they are doing by reading a paper on their site.
>>But being a tax lawyer my mathemetics bothered me. Can someone please explain
>>their methode in language that is understandable to some one with only high
>>school mathematics.
>>
>>Thanks
>>
>>Jan Willem
>
>Oh well scientific world is going to kill me (the AI part),
>if i say this, but all those learning things are closest to
>next thing:
>  In a program there are values, for example open file might
>  get 0.20 pawn bonus.
>
>All those bonuses and penalties we call parameters.
>
>Automatic tuning, whatever form more or less is going to randomly
>guess a different value for every parameter.
>
>Then the program plays a few games, or most likely only does a few
>tests. Based upon the result of the test then gets decided whether this
>tuning is better or worse.
>
>Obvious problems
>  - if the way of testing is very primitive, so will the conclusions
>    about a parameter set be
>  - less obvious, but hard reality is that from game playing viewpoint
>    the best versions of diep always were versions which did less well
>    on the testsets. Too aggressive evaluation tuning solves of course
>    way more positions but there is every move perhaps 50% chance that
>    you play a patzer move and lose the game because of that, whereas
>    a more positional alternative is more likely to winning the game
>    chanceless.
>  - No independant conclusions get drawn like humans can.
>    Like we watch a program play and say: "he it is getting way too much
>    points for open files, it goes to the open files quick but neglects
>    the rest of the play!"
>  - So if there is 1 parameter in an evaluation,
>    then the automatic learning needs
>    to try all values for that 1 paramater before it knows whether it's
>    Suppose we have 2001 values for 1 parameter (-1000 to +1000), then
>    that's 2001 experiments. Now for 2 parameters it would need
>    2001 x 2001 experiments.
>    Obviously for 100 parameters it needs 2001^100 then
>    for 1000 it needs 2001^10000. That's a number which will not fit
>    soon on your paper so many zero's it has.
>    etcetera.
>    In short automatic learnings weak point is the huge number of experiments
>    needed. Obviously every researcher in automatic tuning focuses upon
>    methods to get that number of experiments as
>    small as possible.
>
>    Till today however no one succeeded in getting it lineair. To do that
>    you need intelligence and domain dependant knowledge.
>
>    That last must not be underestimated, but the first still gets hugely
>    underestimated by non-scientists.
>
>    The learning program is usually using very simplistic algorithms to
>    get to its numbers.
>
>    The average chessprograms search is usually way smarter as the most
>    complicated learning which is producing results somehow.
>
>    The trick 100% of all researchers use to show their learning works
>    is to compare with the most insane parameter set so called 'handpicked'.
>
>    If i would only tune for a day, then let the automatic tuning of knightcap
>    learn and learn and learn for over a year or 10. Then play
>    knightcap-diepeveen tuned versus knightcap-selftuned,
>    then we would see a huge score difference. Of course if i were
>    the knightcap programmer i would start with a bonus for doubled pawns
>    and a penalty for putting rooks to open files. In that way i could
>    conclude that learning works.
>
>    This is how 100%, can be 99.9% too, of the conclusions are about
>    learning.
>
>    Way too optimistic. It's really a childish thing!
>
>    Now it's unfair to say something bigtime negative about knightcap,
>    because the level of this experiment is already way above other
>    tuning experiments. The average tuning experiment is way smaller setted
>    up, and if such a childish average experiment gets filmed by discovery
>    channel or whatever, then they also directly conclude that in future
>    robots are going to take over.



This page took 0.07 seconds to execute

Last modified: Thu, 07 Jul 11 08:48:38 -0700

Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.