Author: Enrique Irazoqui
Date: 14:23:17 03/17/98
Go up one level in this thread
On March 17, 1998 at 12:08:00, Ed Schröder wrote: >>Posted by Enrique Irazoqui on March 16, 1998 at 19:34:31: > >About F5-200 <> R8-90 31.5 - 8.5 > >Enrique wrote... >The problem is that this partcular result is not realistic. > >Ed Schroder wrote... >Here you say it. > >Enrique wrote... >Same reason: flawed learner. > >So you admit that "learners" give "not realistic" results? This is sophistic. I agree that programs with badly designed learners will be at a disadvantage against programs with better learners. >Do you agree with me that without a decent learner a chess program >on SSDF is killed these days? Sure. >>Not what I heard from other programmers. Not what I have seen in new >>programs with good learners. They just don't lose the same game twice. > >You are overlooking one very important and crucial fact. >See below. > > >>>Moreover you indirectly say: "The best learner wins the jackpot".... > >>The contrary. What I m saying is: with good learners, and they do exist >>today, this whole issue is irrelevant. It was important a few years ago, >>when learners didn't exist. Now this prehistory. > >Wrong view. >See below. > > >>We all want to know the real strength of engines. A good learner does >>the trick. > >Sure and learners do a lot more. > >As you said unrealistic results if the computer opponent: > >#1. has no learner at all. >#2. has not an aggressive learner. > >I have tested this myself with Rebel9. Played a series of games against >program_X. Program_X had NO learner. Results (by head) > >Rebel9 (learner off) - Program_X 24-16 >Rebel9 (learner on) - Program_X 33-7 > >The 33-7 is a cheat. The 33-7 doesn't reflect the real playing strength >between Rebel9 and Program_X where the 24-16 result does! Of course the >33-7 had many many doubles because the Rebel9 learner simply starts >repeating the games he previous already had won. That is a cheat. > >Second example.... > >Tested Rebel9 against Program_Y. Program_Y has a learner! Results >again by head. > >Rebel9 (learner off) - Program_Y (learner off) 22-18 >Rebel9 (learner off) - Program_Y (learner on) 20-20 > >Rebel9 (learner on) - Program_Y (learner off) 29-11 >Rebel9 (learner on) - Program_Y (learner on) 27-13 > >This shows there is a huge difference in the learning quality >between these 2 programs. In no way this does reflect the playing >strength between Rebel9 and Program_Y. > >The only reliable result is the first one (22-18) the one with >no learners. Now you can blame Program_Y for not having a good >learner. My point is that with comp-comp learning you can gain >a lot elo points on SSDF. This is a cheat. Not anymore. We are not anymore testing engines. We are testing programs as a whole ever since auto232 appeared. These results you post above would have been much more even in the case A and B had a similarly competent learner. You could say the same about opening books. Test program A with a tuned book and program B without a tuned book. Results won't be realistic either. Example: Mchess5-Rebel6 and Mchess5-Rebel8. Most of the huge difference comes from the book. That's why programmers spend ages tuning books. They even hire people to do it. Is this a cheat? No. It is an integral part of chess, just like learning is. Does this measure the engines? In no way. And nevertheless you don't ask for engines to be released without books. >Now to your theory that learning is so easy to program. It is not. >Learners are so easy to mislead. > >Example 3.... > >Program_Z has a good (but hidden) learner. You can't put it on or off >it is just always on. I played hundreds of auto232 games to test the >impact of the Program_Z learner. It goes like this.... > >Rebel9 (learner off) - Program_Z 24-6 round-1 >Rebel9 (learner off) - Program_Z 20-10 round-2 >Rebel9 (learner off) - Program_Z 16-14 round-3 >Rebel9 (learner off) - Program_Z 14-16 round-4 >Rebel9 (learner off) - Program_Z 12-18 round-5 > >Program_Z is heavily learned against Rebel9. > >But now comes the trick. I let Program_Z play 200 games against another >program. Which? I forgot but it doesn't matter. Program_Z starts to >learn against this program. > >Now I repeat the Rebel9 testing and Program_Z is totally confused by >the 200 games he just played. Rebel9 simply wins again against Program_Z >with remarkable scores. > >Silly isn't it? > >So take my word for it that comp-comp learning is still in childhood >stage. So much to improve. And comp-comp learning is simply a cheat >as the goal is to get a higher elo on SSDF. It has nothing to do with >the strength of the chess engine. As above, neither do tuned books, also made with the SSDF in mind. >Blame me for my part in it. With Rebel9 I joined the club. Now I >step out. It was a mistake. I will not support this silly cooking >race any longer. Back to the roots which is the chess engine. I still think this focusing in engine alone is prehistory. Programs play games from the first move to the last. They all count: from openings with tuned books and learners to endgame tablebases, and everything in between. Not only the engine. Learners are here to stay. You can curse them or you can say they are most welcome: avoiding cooked lines, avoiding losing the same game 100 times, useful also in computer-human playing. Maybe they are difficult to program. So is chess itself. Besides, Ed, there is no way around it. Programs do have learners. If you step out, you step out of the competition. What a shame. Enrique >- Ed -
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.