Author: Vincent Diepeveen
Date: 11:50:20 10/10/00
Go up one level in this thread
On October 10, 2000 at 10:07:03, Graham Laight wrote: >On October 10, 2000 at 08:36:10, Vincent Diepeveen wrote: > >>On October 10, 2000 at 07:05:45, Graham Laight wrote: >> >>>It seems to me that PCs' results against GMs are tapering off into a flat line. >>>The current style of program may have come as far as they can go. >>> >>>The battle to generate the highest NPS score is no longer improving the >>>computers' performance against humans. Even Deep Junior running on a quad >>>processor is only able to score 4.5/9 against the top players. >>> >>>With dozens of programmers competing to make the "final push" to get programs >>>ahead of humans, to impartial observers it looks like the harder they push, the >>>more the bandwagon gets stuck in the mud. >>> >>>Programmers also have to remove knowledge from their eval fns to score higher >>>against their computer opponents. >>> >>>Looks like a doubling of NPS no longer provides an extra 50 Elo rating against >>>humans - nothing even close, in fact. >>> >>>I wonder whether some of our common assumptions about how speed and knowledge >>>affect ELO ratings are wrong. >>> >>>I think we mostly assume that the return on both knowledge and speed look like >>>the picture below: >>> >>> | >>> | >>> | ** >>> | ************************** >>> | ** >>> ELO | * >>> | * >>> | * >>> | * >>> | * >>> | * >>> | * >>> |* >>> |* >>> |* >>> |--------------------------------------------------------------- >>> Either Speed or Knowledge >> >>remove knowledge from this axis. >>not many very knowledgeable programs so far have had a chance to >>compete in big tournaments or in big matches. > >Agreed - the shape of the graph is proven for neither knowledge nor speed. i would sure want to say that there is clear proof for the speed side. more knowledge as the average testset used to proof a new algorithm in JICCA. >It just seems as if each time the top level plateaus, it stays at the plateau >longer than last time. And possibly jumps upwards less when it starts to move >again. though this looks to me a logical assumption the granularity definitely is so far defined by commercial challenges. obviously those challenges are done by companies who want their investments seen paid back, to their viewpoint is not scientific. >>> >>>But what if, in reality, one or both of them actually looked like this? >>> >>> | * >>> | * >>> | * >>> | * >>> | ** >>> | * >>> | * >>> | * >>> | * >>> | * >>> | ** >>> | * >>> | * >>> | * >>> | * >>> | * >>> | * >>> | ** >>> | * >>> | * >>> | * >>> | * >>> | * >>> ELO | * >>> | * >>> | ** >>> | * >>> | * >>> |* >>> |* >>> |--------------------------------------------------------------- >>> Either Speed or Knowledge >> >>>In other words, shooting up, plateauing for a while, then shooting up again - >>>and so on. It's possible that, because chess programmers vary the amount of >>>expertise between 20 and (say) 500 distinct pieces of knowledge, they've found a >>>plateau (probably the 2nd one), and, angry about being beaten by someone with >>>less knowledge but higher NPS, have refused to go down the knowledge route >>>seriously. Also, from many years of reading postings in this group, it is >>>apparent that NPS, and techniques to raise it, is where the focus lies with this >>>particular group of people. >> >>obviously fritz3 would search 17 ply at nowadays hardware if it was >>updated a bit to 32 bits + 8 bits code and hashtable would get implemented >>a bit better. Yet fritz6 doesn't get that. > >I have to disagree with this. From my understanding: > >* Fritz 3 used root processing to achieve its speed. But root processing at 17 >ply would be completely useless - so lowering the NPS per MHz is inevitable. for the depth versus playing strength graph this is not relevant, idem for the arguments below. However for the more knowledge + deeper search versus performed rating i definitely think the fact that all kind of programs get rewritten is showing a clear thing. So far it's not clear to what junior gets rewritten to, but as soon as Amir wants to talk about that, we might have more datapoints. >* Fritz almost certainly has more knowledge now - but the knowledge won't have >increased by the same order of magnitude as the computers' processors. The >computers are at least 20x faster now (10x clock speed, processors doing at >least twice as much useful work per clock cycle). Has Fritz 6 got 20x the >knowledge of Fritz 3? My guess is that it hasn't. It probably had about 100 >distinct pieces of knowledge then, and probably has about 300 now (excluding frans talked about fritz5 needing 200 clocks to evaluate. hard to fit in 100 distinct things which apply to the same type of position. >tablebases). Like all the other speed focused programmers, Franz has trouble >integrating new knowledge with the existing knowledge. i think frans does a good job here. it's obvious that in newer version fritz plays a lot more patzer moves, see the sac against crafty in wmcc and comparing scores from 5 and 6 clearly shows difference. >If you want to manage (a huge increase in) knowledge better, you might be better >off using knowledge management techiques (which I have discussed in CCC before). 'which i have discussed in CCC'. i would like to add something to diep which can redenate about all patterns, so a higher language form of evaluation. Yet my program would slowdown 3 times or so at least, that's quite a lot, implementing it in neat c++ even with fast techniques would be perhaps 10 times slowdown, and that all for a few hundreds of a pawn score at most, as with meta knowledge you gotta be careful. Now the biggest problem is the huge work it is to rewrite everything AND keep the current fast evaluation structure. Now where i already have all kind of expensive loops, the basic concept in fritz can't be hard. suppose frans has in fritz next: if( pattern ) then evaluate. Now that's real slow. So he can only add constants or needs to rewrite without the 'if then else' clauses or he slows down too much. in 200 clocks even the best assembly programmers can only do a few things. So there is still a lot for him to improve to programs like mine. note that his tuning of parameters obviously is miles ahead of mine. if you score so well as fritz does, then obviously it's testing is quite clever. Search no one even discusses about. That was always brilliant. >However, using these techniques will certainly entail a big sacrifice in speed. that is not bad. using attacktables, mobility and a lot of different patterns, make my program in total like 20 times slower as Fritz is. Note that the good thing is that newer generation of processors and memory is favouring me bigtime. >Might just be a better use of multiple processors (and in the future, a computer >might have thousands of them), however, than the diminishing returns which can >be obtained from using these processors for tree search. the whole diminishing returns theory doesn't take a lot of factors into account, not to mention realism. It's obvious that shredder wins tournament after tournament with superior endgame and good testing. chessbase products usually are equipped with the latest too. latest book for example. without wanting to say that this is a bad thing (as it's still not near human standards, so hard to say that using this book is bad), it definitely matters against GMs. As a lot of those factors are completely different then. So far one could play chessplayers without caring much for opening or far endgame. Most likely he blunders away a piece somewhere and its finished. Now it suddenly appears that if GMs take opponent serious (as a few players did against junior for example), without even showing their opening still, they can produce a position where they can profit from weaknesses of the thing. So far that hardly had been done. if it was done, then they all played like they would play either fritz or crafty. These human kind of factors hardly get taken into account so far. "did he really try to win or draw, or did he just didn't care?" it's obvious that kasparov played the weakest games of his life (including games when he was 18 years old and such) against deep blue. it's obvious that in blitz at the internet most gm's are practicing and not too much out for the point. it's obvious that some gm's still have hard problems in openingspreparement to see the difference between an objective won position and subjective won position (where good tactics can win the position, so where computer might simple win the position where human fails to win it) on the other hand at the highest level such kind of games are not played much. Personally i think Shirov has a big tendency to play a bit less objective chess. Without question and doubt however at the same time thereby saying that he also is real good in figuring out what to play to show weaknesses in computers (see how shirov wins at 2 0 even against computers!). Yet many stronger GMs are more busy with a much higher level of chess in the opening and usually more objective chess as so far programs have faced. it's hard to call the openings kasparov played against deep blue his 'normal' openings for example. as a moron he played openings which only get played by karpov. obviously kasparov failed in even recognizing positions where he could have profitted bigtime, as pointed out bigtime by Seirawan in JICCA june 1997. Seirawan without shame gives on average like 5 moves the dubious or even bad move sign (?! and ?) from kasparov. Normally giving a single move of Kasparov a question mark in a game he played already is enough to get executed in all magazines by extensive analysis. So if Seirawan would have commented games of kasparov in a tournament like how he commented on the kasparov versus deep blue games (so many question marks), then Seirawan would have been shot dead, sent to Mars; but Yasser never could have asked to return to planet earth without fighting Kasparov in a life or death battle. Yet Seirawan got away with it, only a smart part of the world must have noticed it. Regurarly at tournaments i play a game kasparov-deep blue a few moves and ask a grandmaster or IM his opinion when he has a bit of time left to analyze. To my big surprise the positiosn don't get recognized. however all games from matches kasparov-karpov most know by head. At least they have played them or they recognize it: "wasn't this a game kasparov once played?" The few who did analyze the games have no good word for it. Such things the graphs don't take into account however. So the achievement of junior definitely is a much bigger one as DB ever did. >-g > >>So obviously search depth is not the holy thing to achieve. >> >>Now that in most games the programs search >= 10 ply it's obvious that >>better knowledge comes first. >> >>Less and less commercial programs are pruning using dubious methods like >>FHR, evaluations get less lazy, and less moves get pruned on alpha in >>the qsearches. >> >>It's all obvious that a more accurate score is preferred nowadays above >>tactical search depth. >> >>Yet the big impact was of course the nullmove, which with the introduction >>of bigger processors kicks butt bigtime. Yet at the same time as starting >>to use nullmove i notice in diep (don't know whether other notice that too) >>that if it's used with last ply pruning in one or another way, that >>there is a clear loss of knowledge, as it takes at least another few ply >>to get the same move as best move just being seen as 0.004 better sometimes >>as the previous best move which in fact is not that a good move compared >>to the newly selected move. >> >>Also hashtables influence a lot. Even small search depths like up to 9 ply, >>my diep needs 2.5 times more nodes WITHOUT transposition + best move storage >>in obvious to sort positions! >> >>>Could it be that, for all these years, we've allowed ourselves to be fooled by >>>the apparent "computer friendliness" of chess, followed the same path >>>progressively, and neglected to address the problems with knowledge coding, >>>knowledge selection (during the game), and knowledge acquisition - which are >>>themselves very interesting computing problems? >> >>>Clinching evidence: humans are still able to beat computers. They aren't using >>>NPS, therefore they must be using knowledge. >> >>Could have been my conclusion, but i would never manage to write it down >>that polite! >> >>>-g
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.