Author: Enrique Irazoqui
Date: 16:45:08 03/17/98
Go up one level in this thread
On March 17, 1998 at 19:04:52, Ed Schröder wrote: >>Posted by Enrique Irazoqui on March 17, 1998 at 17:23:17: > >>>I have tested this myself with Rebel9. Played a series of games against >>>program_X. Program_X had NO learner. Results (by head) > >>>Rebel9 (learner off) - Program_X 24-16 >>>Rebel9 (learner on) - Program_X 33-7 > >>>The 33-7 is a cheat. The 33-7 doesn't reflect the real playing strength >>>between Rebel9 and Program_X where the 24-16 result does! Of course the >>>33-7 had many many doubles because the Rebel9 learner simply starts >>>repeating the games he previous already had won. That is a cheat. > >No comments here? I commented this before. OK, I'll repeat it. The above is not real when both programs have a good learner. >It's a cheat no? No. Again: a decent learner in both will let the engine make the difference. Not such a sophisticated learner, by the way. A simple one, a human-like one. A learner that repeats the opening line when winning and avoids it when losing, just as we do. In the process, it takes care of double games. It doesn't seem so difficult to do. Amir says is an overnight job. Crafty has it, Fritz has it... >>>Second example.... > >>>Tested Rebel9 against Program_Y. Program_Y has a learner! Results >>>again by head. > >>>Rebel9 (learner off) - Program_Y (learner off) 22-18 >>>Rebel9 (learner off) - Program_Y (learner on) 20-20 > >>>Rebel9 (learner on) - Program_Y (learner off) 29-11 >>>Rebel9 (learner on) - Program_Y (learner on) 27-13 > >>>This shows there is a huge difference in the learning quality >>>between these 2 programs. In no way this does reflect the playing >>>strength between Rebel9 and Program_Y. > >>>The only reliable result is the first one (22-18) the one with >>>no learners. Now you can blame Program_Y for not having a good >>>learner. My point is that with comp-comp learning you can gain >>>a lot elo points on SSDF. This is a cheat. > >>Not anymore. We are not anymore testing engines. We are testing programs >>as a whole ever since auto232 appeared. > >Not true. > >SSDF still test the newest programs (with learners) against older >program with no learners or poor learners. Result, incredible scores >because of the advantage of the (aggressive) learner. Just check >the SSDF list and notice these many crazy match results which do >not represent the real playing strength difference. > >In NO way does these scores represent the real playing strength >between the newer and older program. > >True? True. The gap is or will be unreal for a while. So what. Or you prefer to bridge the gap by killing the list? >>These results you post above would have been much more even in the case >>A and B had a similarly competent learner. > >Exactly the point!! > >I want to see clean fights between 2 engines and not a fight between >2 smart learners. Isn't the engine-engine fight not the original goal? And how do you achieve this today? With the damn learners! I have played hundreds of games this last few months with the latest generation programs. In only one instance I have seen a program falling twice for the same losing line. Where is the problem? Ah, yes. With older programs without learners. Not that terribly important. And by far the lesser of evils, since the solution you propose is quitting computer-computer altogether. >Also why do you think people buy a chess program? What do you think >is more valuable for the customer who wants to purchase, the engine >or the SSDF COMP-COMP learner? What about all of it, engine+learner+book+endgame tables? >>You could say the same about opening books. >>Test program A with a tuned book and program B without a tuned book. >>Results won't be realistic either. Example: Mchess5-Rebel6 and >>Mchess5-Rebel8. Most of the huge difference comes from the book. That's >>why programmers spend ages tuning books. They even hire people to do it. >>Is this a cheat? No. It is an integral part of chess, just like learning >>is. Does this measure the engines? In no way. And nevertheless you don't >>ask for engines to be released without books. > >You know how I think about book preparation. >Never did. I don't mean cooks. I mean checking the lines Rebel plays best and worst. Checking it with other programs. Collecting computer games to see just this. >>>Now to your theory that learning is so easy to program. It is not. >>>Learners are so easy to mislead. > >>>Example 3.... > >>>Program_Z has a good (but hidden) learner. You can't put it on or off >>>it is just always on. I played hundreds of auto232 games to test the >>>impact of the Program_Z learner. It goes like this.... >> >>>Rebel9 (learner off) - Program_Z 24-6 round-1 >>>Rebel9 (learner off) - Program_Z 20-10 round-2 >>>Rebel9 (learner off) - Program_Z 16-14 round-3 >>>Rebel9 (learner off) - Program_Z 14-16 round-4 >>>Rebel9 (learner off) - Program_Z 12-18 round-5 > >>>Program_Z is heavily learned against Rebel9. > >>>But now comes the trick. I let Program_Z play 200 games against another >>>program. Which? I forgot but it doesn't matter. Program_Z starts to >>>learn against this program. > >>>Now I repeat the Rebel9 testing and Program_Z is totally confused by >>>the 200 games he just played. Rebel9 simply wins again against Program_Z >>>with remarkable scores. > >>>Silly isn't it? > >No comments here also? Exactly the same as before. See above. >>>So take my word for it that comp-comp learning is still in childhood >>>stage. So much to improve. And comp-comp learning is simply a cheat >>>as the goal is to get a higher elo on SSDF. It has nothing to do with >>>the strength of the chess engine. > >>As above, neither do tuned books, also made with the SSDF in mind. > >True and explained by Jeroen. >Again Jeroen never did. Jeroen never collected games played between Rebel and other programs in order to tune the book for openings Rebel plays best? Again, I am not talking about cooks. >>>Blame me for my part in it. With Rebel9 I joined the club. Now I >>>step out. It was a mistake. I will not support this silly cooking >>>race any longer. Back to the roots which is the chess engine. > >>I still think this focusing in engine alone is prehistory. Programs play >>games from the first move to the last. They all count: from openings >>with tuned books and learners to endgame tablebases, and everything in >>between. Not only the engine. Learners are here to stay. You can curse >>them or you can say they are most welcome: avoiding cooked lines, >>avoiding losing the same game 100 times, useful also in computer-human >>playing. > >Actually saying, go Ed and cook your opponents as best as possible >on SSDF. No Sir... You know I never implied that. Cooks is not the word. Once more: picking a line you play best is not cooking. Avoiding a line you play badly is not cooking. Rather clever, as a matter of fact. And it is ALL it takes to avoid the problems you have with the SSDF games. If all programs have this learner, and most already do, the problem is gone. >>Maybe they are difficult to program. So is chess itself. Besides, Ed, >>there is no way around it. Programs do have learners. If you step out, >>you step out of the competition. What a shame. > >Sure there is a way out. > >A Rebel is a Rebel. The Ajax is the Ajax. Because it's good AND participates in competitions. Enrique >- Ed - > > >>Enrique
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.