Author: martin fierz
Date: 17:27:10 03/27/02
Go up one level in this thread
On March 27, 2002 at 17:05:09, Dan Andersson wrote: >It might be so. But huffman decoding can be speeded up quite a bit by using a >tad more memory for a decoding cache and lookup table. As long as they fit in >the CPU cache it might be possible. But if it's feasible or not depend on A) the >different compression ratios B) the penalty for memory accesses. Where A will >depend on the nature of the data. And B will depend hevily on A. Modern CPUs >have far outstripped the ability of main memory to supply them with data, and >HDs are dog slow. Therefore what was true a few years ago must be re-examined. A >hybrid scheme might be envisioned. (All depending on the data compressed.) Where >the HD files might use a heavier compression. (There are systems that manage to >compile source code (or intermediate representations) and start the programs >faster than it could load a pre compiled program. Due to the greater size of the >executable) >And the gain in loading time could be used to do a decompress to RLE or even >Huff code in memory. And the resulting memory block might then be further >decoded in the cache due to the possible gain due to the smaller amout of memory >transferred. Such a multi tiered approach might be possible, and maybe even able >to supply the CPU with the data it needs. But the tuning and scheduling of it >might be a nightmare. Or a pure joy. This is all fever driven speculations from >me. And I know just enough compression theory to be dangerous :) But the main >point is that algorithmic approaces that were unfeasible a few years ago. Might >fit better in todays or tomorrows architectures. Boy, this is interesting! > >MvH Dan Andersson you definitely have a point here. not only because of everything you say, but also because the 8-piece database, compressed by the chinook team, is something like 5GB in size. i haven't quite finished the 8-piece database, but my 6-piece db is about 10% smaller than theirs. i also have a small improvement to my algorithm which could give some more compression. anyway, the point is of course that even if it's "only" 4GB in the end, you can never load it into memory, and always hit the harddisk. under windows, you can get 2GB, but no more, so if you could fit the whole thing in 2GB, you might be able to access it much faster even though decompression takes much longer CPU-wise - as long as you don't hit the disk it wouldn't really matter. unfortunately, i'm preparing for a world championship challenger tournament, and don't have too much time to look deeper into this :-( and will stick with the simple RLE. i also just saw that the chinook guys are publishing their 8-piece db finally, so maybe i'll just throw it all away... aloha martin
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.