Author: Stuart Cracraft
Date: 12:55:58 07/10/04
Go up one level in this thread
On July 10, 2004 at 12:07:40, Robert Hyatt wrote: >On July 10, 2004 at 11:20:46, Stuart Cracraft wrote: > >>On July 09, 2004 at 08:52:25, Robert Hyatt wrote: >> >>>On July 09, 2004 at 07:16:41, Volker Böhm wrote: >>> >>>>On July 08, 2004 at 11:44:32, Robert Hyatt wrote: >>>> >>>>> >>>>> >>>>>One fatal flaw. This will produce a set of values that will optimize your >>>>>results against a test set. But that's not the same thing as producing a set of >>>>>values that will optimize your results in actual OTB games. >>>>> >>>>>This is a common mistake. >>>> >>>>Have you got an idea how to automatically optimize settings for OTB games? >>> >>>I have tried the test set route. It simply didn't work. I have altered the >>>settings to produce better results on ICC, and hurt test set results. My search >>>parameters are tunable by simple commands to crafty, so it is easy to automate a >>>big test set and vary each parameter over some range. There was definitely one >>>or two overall combinations that were best for test positions, but not for OTB >>>play... >>> >>> >>>> >>>>Currently I need about 2 days on two computers to test if one setting is better >>>>than another (with acceptable low error rate). The following experiences makes >>>>things hard: >>>> >>>>1. You can only find settings that are at least 5% better (gets 5% more points) >>>>by testing. Optimizations below 5% will need to much games to give a >>>>statistically "proven" (I use "it is better with a probability of 95% or more") >>>>result. >>>>2. Even for a "5%" better you need about 200 test-games. >>>>3. The result will differ with different time-control. I ignore this problem >>>>currently. >>> >>>There are several setup parameters that will cause problems. You just pointed >>>out one, the time control. Blitz is different from longer games and the >>>parameters will likely be different. Of course this is an important "detail" >>>that an engine should deal with, but I do not myself. But I can see where it >>>would be good to tune for the time control, somewhat like my "adaptive hash" >>>tunes the hash size for the time control, automatically... >>> >>>You also can't ignore the opponent. I'd expect to find different parameter sets >>>for different opponents would be more optimal as well. >>> >>> >>> >>> >>>>4. The result will differ with different opponents. I user a set of opponents. >>>> >>>>Thus optimization is really hard work for me! >>>> >>>>Greetings Volker >>> >>> >>> >>>It's hard for everyone. :) >> >>Automate it! >> >>Make everything in your program easily changeable by the program >>itself -- so everything is a variable indexable by a single number >>and that expands out to an entirely different opponent. > >Already done... > Outstanding! >> >>Then let it loose on ICS/FICS with some method to keep those values >>that contribute to good moves and wins and discard or change those >>values that don't. Many methods for autotuning exist. It's been pretty well >>understood that in chess using just the end-of-game result won't help tune it >>very quickly as compared to per-move results. > > >That's the problem. Figuring out which are good and which are bad... > Have you seen Whitwell and Kendall's paper on Evolutionary Computing? > >> >>Thus, Knightcap's TD(leaf) and Andrea Nowatzyk's grandmaster-game-tuning with >>Deep Thought are the auto-tuning methods of choice, at least that I've >>heard of -- but I don't hear about them being done very often. Research >>vehicles that fall into obscurity? > > >Been testing a similar approach for a while. Anthony Cozzie has written some >code to do this. But while it sounds easy, in practice it is far from it... > I didn't think it would be easy, otherwise why hasn't it been done more? Like bitboards. Took awhile for those too. :-) > >> >>Autotuning as far as I've heard and read has produced results as good >>as hand-tuned with a lot less heartache. With ICS and FICS being the >>proving grounds par excellence, I assume this method must be more widely >>used than I've heard. > >I have _never_ seen an auto-tuned program that was as good as a hand-tuned >program myself. The work Anthony has been doing has not reached this point yet >although it certainly might. The problem is the "tuning set". If it doesn't >represent real game positions the program will see, then it is useless... > The comparison is the problem and the amount of work. Apples and oranges after all. The tuning set obviously needs to be the real games and I saw the value of ICS/FICS many moons ago as well. Interesting lab. Stuart
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.