Author: Robert Hyatt
Date: 12:49:45 12/03/02
Go up one level in this thread
On December 03, 2002 at 12:54:24, Rolf Tueschen wrote: >Until now nobody out of the programmer group had ever spoken about that evident >truth. SMK says that these tests can't show the strength of play or as it was >claimed for this test, the "ability to analyse". SMK also explained (for the >first time in that direct speech) how he and every programmer could fake the >results of such tests. He then speaks about the question if it could be >discovered, as it was by T. Mally in case of Ed Schröder, and he saud that of >course he could do it so that nobody could find out. In fact he had written such >a "tool", but in the end he decided to let it out of the commercial product. This is a half-truth. Test sets can and do measure tactical skills. But that is only a part of the game. You also have to play moves when there are no tactics to deal with, and that part of the game is uniquely unsuitable for most test sets. However, too many rely on test sets as an "absolute truth" and that leads to flawed or completely wrong conclusions... That said, I would disagree with the idea that there is no way to test positional knowledge with a test position. That is simply wrong. IE if you want to see if a program understands (a) having a distant pawn majority is good and (b) if you have one, the less material left on the board the stronger it becomes, then it is possible to create a position that will reveal that the program clearly does _not_ understand the idea, although it is nearly impossible to create a position that shows that it _does_ understand the issue. IE programs are pretty good at making the right move for the wrong reasons. But it is very difficult to make the wrong move for the right reasons since that is a contradiction. :) > >But all this gives me the opportunity to talk about the reasons why such a >testing with even these top class positions is nonsense. And why it has nothing >to do, well, almost nothing, with _real_ strength. > >I think I can show you why in special for those allegedly positional positions >the test is nonsense and that he's measuring something else, but not analysing >power of the engine. > It is mainly measuring (a) the tactical ability of the engine; (b) the aggressiveness of the engine's evaluation. Because most test positions involve something to do with winning material, or exposing the king to a violent attack, or producing some easy to see endgame winning idea... >I will keep it very short so that you can do your own research. > >(Just to mention that I asked for that problem already two years ago as >'Schachfan' in CSS forum, but then it went about a tactical mate position). > >Look, if you have a positional game of chess, where do you choose the point for >a test? Of course, in this WM-Test of Gurevich et al you take the position when >exactly a certain by the experts well commented and mostly beautiful move has >been made. Because there the commentators said: only with this move he could >conservate the slight advantage.But the truth is that often the engines see - in >their actually possible realm - two solutions very closely together. And in >general it could be said that for positional positions without tactics the evals >are not very impressive at all. So, how could you calculate it in your results? >Would you really take a difference of 0.01 points as decisive? Is that relevant? > >But the main problem of such test positions is this. > >The point of that "nice move" (that caught th attention of the commentators) is >by no means the most important moment for the decision making. Let me explain >the irony. The usual commentators are masters themselves. Well, and therefore >they take certain decisions as completely normal, because they are easy and >trivial for _them_, but not so for the amateurs. Or the machines so to speak. >But now go with me bachwards a few moves. How optimistic you are that we could >then expect that a machine would be better prepared to make the right decision >in such _positional_ games? And that is exactly the point for these test >positions. _Realistically_ we had to test the machines in positions, where only >experienced humans know how to play to be later in the position to make some >"decisive" moves, moves then commented by our experts. Only the early positions >would allow a verdict if our actual machines could play posiional chess. We know >already the answer. They can't for the moment. > >But therefore such tests with such a great pretension are a fake, a hoax in >themselves. And Stefan MK explained it with the possible distinction. In reality >M. Gurevich is making a question of life or death out of it. But earlier >somewhere I already mentioned that it's ridiculous to claim the honor for a so >called, guess that, I translate, World Champion Test. These positions are simply >taken from Wch matches. What a thrill! But it's known for ages that the chess of >these matches is not always the best possible. Because it's mainly a >psychological fight. And fortunately Gurevich didn't claim that he were testing >psychology. But just now it was published that one position wasn't from Wch >chess at all. A game between Anand and Shirov. And to make the scandal even >greater. The authors used a false position. Instead the K stood on c7, they put >him on d7. But with Kc7 we have two solutions. The searched Ng5 and now the odd >Bg5 too. Christ! A whole life work of a few hours of choosing some positions out >of Wch games is in danger to lose all reputation. Doctor doctor, gimmi the >news...! > >Rolf Tueschen > Anyone that believes ratings produced by test sets is simply naive. Anyone that believes that a test set can prove that a program is playing at a world-champion level is simply stupid. :) >On December 03, 2002 at 09:26:42, Eduard Nemeth wrote: > >>Very interesting post from SMK in CSS Forum (only german). >> >>Please read it, i thing that a translation is interesting for You! >> >>Read here: >> >>http://f23.parsimony.net/forum50826/messages/54995.htm
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.