Author: Adrien Regimbald
Date: 05:15:58 06/19/00
Go up one level in this thread
Allo, >I also loop through all games/moves, writing out each position to a binary >hashed database (I use the GDBM library). Once I'm done looping the games, >I loop through all positions in the database and cull out everything that >isnt't played at least 3 times. > >Somed data: a 34MB PGN database of PGN games leads to a 47MB database on >the first pass, which is reduced to 3.7MB after the second pass. > >Crafty's book for the same games and with the same parameters is only 1.3M. > >I wonder how this is possible, considering that I don't store as much data >per position. (only a times played long int). Maybe GDBM isn't as efficient >as it claims to be? Probably you pay some overhead for the ease-of-access .. and probably also it is not perfect at allocating how much space it needs to hold everything (ie. there are probably some empty hashing cells that still get stored into your final output book). I really wouldn't know though, I'm not familiar with GDBM .. but from past experience with other hashing tools to make your life simpler, they usually aren't efficient .. but they DO save you a substantial headache! :P >My main problem right now is speed...it takes redicolously long to parse in >all PGN (darn my parser is slow) and GDB slows down due to disk accesses when >the database gets big (still smaller than physical RAM though...so I dont >get why it does). How long? My book creation took about 5 seconds for those ~13,000 games (a 7 MB PGN file) on a P233 w/ 64 MB RAM. >If you are in need of webspace, I can give you the password for Sjeng's ftp/web >space on sourceforge and you can put it up there. It's a very fast server and >you can have up to 200Megs. Hmm, that'd be cool :) Would it be possible for me to get my own sourceforge workspace (or whatever you want to call it) ?? >If you have no objections to it, I'll incorporate any usefull improvements >into Sjeng too. Of course I don't object! :) In fact, I sent you a copy of the source and everything just before I read this message :P I'd tell everyone else everything I said in that e-mail but I'm sick of typing, and I don't want to just copy/paste the whole thing here ;P Everyone else can see it all later this week ;) Regards, Adrien.
This page took 0.01 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.