Author: Christophe Theron
Date: 22:03:04 12/27/97
Go up one level in this thread
On December 26, 1997 at 16:25:57, Don Dailey wrote:
>On December 26, 1997 at 14:27:44, Christophe Theron wrote:
>
>>
>>On December 26, 1997 at 12:26:23, Don Dailey wrote:
>>
>>>Perhaps they are doing heavy pruning on the computers moves? If you
>>>modify the selectivity such that much heavier pruning takes place
>>>on the computers side you might arrarnge extra depth that results in
>>>play that is safe but not opportunistic for the computer.
>>>
>>>I remember Richard Langs programs USED to have the characteristic
>>>that they would see anything you could do to them. In some experiments
>>>me and Larry Kaufman did, it would quickly see that you could win
>>>a piece and avoid the loss. But if you forced the piece losing move
>>>and let it think for the winning side, it could not find the win without
>>>a very long think! Very strange. But this was the best program in the
>>>world and pehaps still is.
>>>
>>>But this makes some sense to me. I don't think a single chess game has
>>>ever been won without an error on the losing side. If your program
>>>NEVER made an error (you wish!) it would never lose (unless of course
>>>the opening position is a loss for one side or the other.)
>>
>>I have tried recently to modify Tiger this way. First, I changed the
>>time allocation policy to allow the program to play only when the last
>>move in the main line was a computer's move. This allows the program to
>>detect more deep threats. Second, I made a more agressive selection on
>>Tiger's move. So he could miss a combination for him, but wouldn't miss
>>an opponent' s combination.
>>
>>In average, this new version shows deeper lines (1 ply more).
>>
>>I let the program run a long self play against the previous version
>>(exactly the same program except for the two above changes). The rate of
>>play was 5mn/game on a P100, 4Mb hash.
>>
>>Result: no improvement at all. The score was very close to 50%.
>>
>>In addition, the new version performs poorly on combination tests.
>>
>>I've made several others tests with differents changes, trying to prove
>>the idea was good, but I didn't succeed.
>>
>>Still I believe that the idea of being very cautious and conservative
>>has some value. I think I didn't catch the right way to do it.
>>
>>
>> Christophe
>
>Hi Christophe,
>
>This reminds me so much of my experiments (but not the same issue of
>playing it safe.) I will try an interesting idea, self-test the hell
>out of it and almost always end up very close to 50-50 (but often
>on the low side!) I have been through this cycle a million times!
>But once in a while, BINGO!
>
>And I always come away with the feeling that perhaps I didn't implement
>it quite right!
>
>
>-- Don
Yes. Self-test is a strange thing. I do it since 1 month, and discovered
you have to take the results with care.
The first experiments I did were self-test program A against A (that is
EXACTLY the same program playing both sides). I use a set of predefined
openings, each program playing white, then black in each opening. 10
openings give 20 games for example.
What a surprise: self-playing a program against itself in these
conditions almost never gives the expected 50% result!
What should have been a simple procedure then becomes a nightmare.
First of all, why doesn't the match end in a 50% result??? Answer:
because of timing errors. I use the PC internal timer to measure the
thinking time, and the clock precision is 1/18.2 second (roughly). A
search beginning right after a clock tick is different from the same
search started just BEFORE a clock tick, the difference being the
measured time. So a chess engine could sometimes decide to continue a
search, and sometimes decide to stop it, randomly...
Second point: OK, I understand why the result is not exactly 50%. But
shouldn't it be close to 50%? And how close? Answer: make several
matches between exactly the same programs, and write down the minimum
and maximum score you get.
Ok, good idea. Let's try... Well the results range from 39% to 58% when
I do 20 games matches. When I try 24 games matches, I get 46%-56%
In order to stop the nightmare, I now do 28 games matches (14 openings),
and decide the result is significant if it is below or equal to 40%, or
above or equal to 60%.
The problem is that a change has to be quite good to win a match by 60%
or more! If it is a slight improvement, chances are you never notice it
by self-play! But I don't know any better way, because I still want my
self-test sessions to be reasonnably short enough to be able to test
many ideas. A 28 games match at blitz rate usually takes more than 4
hours. So in 1 day, I can only make 3 tests... I use a P100 computer to
do self-test 24h/day, and a K5-100 computer to program. I use self-test
to verify tactical ideas only. I didn't try yet to test positional
evaluation changes, and I'm afraid to see what could happen in that
case.
Last information: run self-test only under DOS (a great advantage of the
DOS programs), because under Windows the error rate in the match result
induced by timing errors is far bigger than 10%!
I would be pleased to read funny self-test stories from all the other
programmers too!
Christophe
This page took 0.02 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.