Author: Robert Hyatt
Date: 11:12:08 06/21/99
Go up one level in this thread
On June 21, 1999 at 12:41:38, Peter Fendrich wrote: >On June 21, 1999 at 09:29:53, Robert Hyatt wrote: > >>On June 21, 1999 at 04:18:57, Jouni Uski wrote: > >Bob, >I mainly agree with you but I think that you give a "one-eyed" view in your >reasons. Because programs and humans have completely different abilities they do >completely different kind of errors. I still agree with you that the GM's >knowledge outperforms the programs tactical abilities, opening lib's, EGTB's >etc, but you don't give the whole picture. >Comments below. > >> >>>After this 2 - 2 (Cilkchess removed) against super GMs with average ELO >>>2621 I think there is no doubt, that PC programs are playing at GM level! >>>Actually hardly no further games are needed. And remember, that average GM has >>>only 2500 as rating. >> >>That is incorrect. 2500 is the _minimum_ rating to earn the GM title, the >>average is well above this. >> >> >> >> >>> >>>I was at the playing hall in Paderborn. The conditions were excellent and >>>sure GMs were playing very serious. I was sitting quite near Fritz - Sokolov >>>board and after 14. Bh6 Sokolov's look was "interesting". He starts to think >>>about 20 minutes. In the continuation I didn't heard him to resign, but after >>>move 24. he simply starts to discuss with Friz operator Feist.... >> >> >> >>Let me give you my reasons why I _still_ consider computers to be 2400 >>players: >> >>1. Pick any of the 5 programs that played the GM players. I will find a >>game where they played so badly that if you look at _that_ game no one would >>consider that program to be a GM. For example, take the winner and look at >>the playoff game. Three different GM players commented that they had _never_ >>seen white screw up the opening so badly... > >True, but take any GM tournament and anlyze all the games with any top program. >They will find a lot of blunders made by the GM's both in the opening and in the >rest of the games. Different abilitites gives different kind of errors. Of >cource GM's find those errors horrible... :) > maybe.. but let's agree on a definition first. GM's make 'single-move blunders" fairly regularly. IE in a game you can almost always find at least one move that has a tactically superior alternative. Most don't lose, in fact most of these would only make the game a bit longer, not change the outcome. Computers play differently. You can find games where anyone would say "Jeez, this looks like an 1800 player on the white side" or whatever... Something that I don't think anyone would ever say about a GM. One 1800-level move in a game? Maybe. Not 10. And not 20. Yet check out some of the incredible binds computers get into at times... And again, not in every game by a wide margin, but more than any GM does... >> >>2. Fritz won in the same way that Deep Blue won the last game vs Kasparov, >>actually even worse... The GM simply made a blunder that a 2000 rated player >>would spot in an instant. If you take that win away, the even result goes >>away. > >With the same reasoning, this is the type of errors human but never programs >does. Even if it's seldom for GM's it happens. > absolutely correct... but it has a strong element of 'chance' in it... IE you might find one such gross blunder in 100 GM games... We are looking at a sample size of 8 so far (5 + 2*Anand + 1*Rohde) >> >>3. I've been working on chess programming for a long time. And regardless of >>how they 'seem' to play in many games, I still know just what they can and can't >>do. And they are nowhere near a GM's level in 'knowledge'. They are still >>surviving on tactics. And there are plenty of GM players that know how to >>squelch tactics and make the game hinge on positional play. And there the >>programs simply don't measure up. > >Not only surviving tactically... Pick your favorite program and talk to the author to see if it understands 'distant passed pawns'. And if so, does it understand 'distant majority' and if so does it understand 'crippled majority' and/or 'immobile majority' and so forth. Things that a GM doesn't even really think about any longer. There are a _bunch_ of such simple ideas that are hard to program, hard to coordinate with other eval terms, etc... And yet without knowing what a majority is, we do well with them because we often see deep enough to convert the majority into a real passed pawn. But the GM recognizes that 'at the tips' while we aren't yet... And the list is quite large... For the longest Roman and Bill Lombardy were doing these 'comparisons' between Crafty and other programs.. and telling me what they saw that Crafty didn't, and vice versa... And I kept trying to explain that the 'differences' didn't matter nearly as much as what _the two GM's_ saw that Crafty didn't, and vice-versa... And when they understood that, their comments became much more helpful, because I was pretty sure that _they_ were seeing the right things while Crafty wasn't... and what the other programs were seeing didn't matter except when it matched the GMs... > >> >>4. I still consider computers to be 2400+. What is the probability that in >>5 games, 2400 players could play evenly with a group of GMs rated 2600? some- >>thing like 25%? > >It should be 20% I think. > >> >>5. GM players exhibit a consistency in quality that computers don't. A >>computer will play like a GM for 5 games, and like a beginner for 1. What >>happens when the GM players learn what the computer can't do and then >>exploit that game after game?. We had at least 4 different GM players watching >>part of the games on ICC (particularly the GM games the last day) and it was >>quite a common question "how can the machine play this opening, and then play >>_that_ move?" Or "why does the machine play so nicely for several games and >>then play like a beginner here?" > >I think this point about GM's (and others) learning how to play against certain >programs is the hardest task to deal with in order to reach GM strength for >programs. The programs inability to adapt to new situations make them very weak >when their weakness is eventually revealed. Opening learning isn't enough here. True... and the only ones that don't understand this are the ones that do not play on ICC. Bruce/I/a few others _live_ there basically. And it is a highly hostile environment. And the GMs are _not_ the problem. The masters/IMs are much worse, because GMs typically try to play "real chess" as they are preparing for an event... an IM is often trying to maximize his rating so that he can sell chess lessons or whatever. And when a good IM starts looking for 'holes' he finds them in _every_ program I have seen. The only thing that saves most programs is that they are 'manual' and the operator will refuse to play after the opponent finds a way to draw every game and win a few on the side... Some manual programs have such notes, in fact. While the automatic programs just have to 'improve' via the programmer to protect against this... I can run Crafty unchanged for two months and I'll guarantee that its rating will start to drop significantly if I don't take time to fix the holes that get exposed... > >> >>So they are very strong, yes. And if you want to lump these games together >>with what we have so far, I don't have a problem with that. So far, I am >>counting: >> >>Humans 5 (counting Anand and Rohde vs Rebel) >>progs 3 >> >>at 40/2hr. That is within reason for a 2400-2600 rating split, particularly >>when the just-finished 5 games were played with little preparation by the GM >>players, since they didn't know who they were playing 24 hours before they >>played. >> >>Yes, I think the machines are doing well. No I don't think they are 2500+ >>_yet_. And the gaps in the program's knowledge are big enough that I don't >>think that tactics alone will make up the difference. It goes a long way, >>but it doesn't close all the holes. >> >>One example (I won't name the GM because he asked me not to, and I won't name >>the program because I am not certain of the hardware nor that the program was >>that which was claimed since I only watched): >> >>The other nite on ICC, a GM friend of mine asked me to watch 2 games a friend >>of his was playing against a computer. The computer hardware was supposedly a >>PIII/500 xeon (1 processor) running a program that everyone here thinks is a >>very 'smart' one based on NPS. In the two games I watched, the GM made it look >>easy. This 'smart' program seemed to not be real aware of the dangers of a >>'distant passed pawn' and allowed the GM to create one, and trade material, >>until it would see it was lost. I wasn't playing the game, obviously, so I >>can't say how 'hard' the GM was 'thinking', but my 'friend' (not the one that >>was playing) commented on this. That is a _fatal_ mistake were this program >>trying to make its way up the GM ranks, because word spreads, and it simply >>gets rolled game after game. I am pretty sure that until a GM finds this >>'hole', that program will give them fits on decent hardware. And I am also >>sure that once a GM finds that hole, it is over except for the occasional >>tactical mistake he will make that the program won't overlook. >> >>The thing we are 'missing' in the programs is the 'consistency' that GM players >>have. IE Shredder played very well throughout the tournament, yet reached a >>position in the playoff that I doubt even a 2000 player would reach. almost >>zero mobility. Cramped. Inactive. A position unlike one I have ever seen >>while watching hundreds of GM games on the chess servers... The problem is >>_not_ impossible to solve. But it is _not_ solved just yet. >> >>This match was 'data'. Let's see how it looks when our data reaches 10 and >>then 20 games. So far we only have 8. And 5-3 is not a horrible result for >>the machines. Unfortunately, based on prior experience, the programs tend to >>win early games and lose later ones as the GM players spot weaknesses that >>can be exploited. Even IM players comment on the current crop of programs not >>doing decently with 'king safety' and how they are all very insensitive to >>long-term kingside attacks. Playing on ICC will expose such. If I didn't >>tweak things regularly, or add code to handle cases I was not handling before, >>I would do horribly on ICC. Because after a while, players find the weaknesses >>and then they sit on those over and over and over. Unfortunately programs are >>complex enough that fixing one often exposes more... > >So here's the race. Wekanesses are exposed making the programs suddenly dropping >in performance, new versions are relesed doing well again, weaknesses are >exposed and so on in a never ending story... >This will not stop, I suppose, in the next 10 years or so... :-) > so it has been forever, so it shall be forever, I suspect, _until_ we get those 32-piece EGTBs. :) >>Other opinions are welcomed of course. This is only _mine_.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.