Author: Dann Corbit
Date: 21:10:11 10/27/05
Go up one level in this thread
On October 27, 2005 at 23:57:24, Dann Corbit wrote:
>On October 27, 2005 at 23:45:08, Yakov Konoval wrote:
>
>>On October 27, 2005 at 22:17:18, Dann Corbit wrote:
>>
>>>
>>>What I would want from a set of EGTB files is the fastest possible access.
>>>So if decomression of the block was slower than reading the decompressed block,
>>>then the compression is literally "a waste of time."
>>>
>>
>>The krbnkrb EGTB has size about 160 GB (with using zlib.dll). With using
>>LZMA-algorithm this size can be reduced approximately up to 100 GB.
>>From the other side the fastest possible access is very important for a chess
>>playing-program, but not so important for data-mining programs.
>>
>>>>If you need provably the best compression, then look no further:
>>>>http://www.ii.uni.wroc.pl/~inikep/research/pasqda41b.zip
>>>>
>>>>According to this document:
>>>>http://www.maximumcompression.com/data/doc.php
>>>>pasqda41b achieved 1.3552 bits per byte against a large body of inputs.
>>>>
>>>>Comes with source code. But expect it to be a slow, slow memory hog.
>>>
>>>I looked up that program (pasqda41b), and it is designed for text files. So it
>>>probably won't work for EGTB files.
>>
>>Usually EGTBs can be compressed much better than 8/1.3552 times.
>>I know PAQAR4 archiver - it really gives optimal compression, but the speed
>>of compression is 4 KB(!!!) / sec.
>>(Probably pasqda41b is written by the authors of paqar4 - I'll check it).
>
>LZMA SDK 4.27 seems an interesting idea to try.
>It is related to the 7-Zip project.
>
>This:
> 144,384 lzma.exe
>compressed into this:
> 61,474 lzma.eze
>42.6% of the former size (too fast to time).
>
>And this:
> 6,810,624 AloneLZMA.ncb
>compressed into this:
> 1,720,055 AloneLZMA.nzb
>25.3% of the former size in 12 seconds for 567,552 bytes/sec
The test program has a benchmark mode, and it seems to decompress at 20
MB/Second.
C:\lzma\SRC\7zip\Compress\LZMA_Alone>lzma b
LZMA 4.27 Copyright (c) 1999-2005 Igor Pavlov 2005-08-07
Compressing Decompressing
1035 KB/s 2409 MIPS 19810 KB/s 2236 MIPS
1034 KB/s 2407 MIPS 19765 KB/s 2231 MIPS
1035 KB/s 2408 MIPS 19766 KB/s 2231 MIPS
1018 KB/s 2370 MIPS 18124 KB/s 2045 MIPS
1003 KB/s 2335 MIPS 19727 KB/s 2226 MIPS
1001 KB/s 2329 MIPS 19680 KB/s 2221 MIPS
999 KB/s 2326 MIPS 19474 KB/s 2198 MIPS
967 KB/s 2250 MIPS 19349 KB/s 2184 MIPS
1001 KB/s 2329 MIPS 18012 KB/s 2033 MIPS
1017 KB/s 2368 MIPS 19822 KB/s 2237 MIPS
---------------------------------------------------
1011 KB/s 2352 MIPS 19330 KB/s 2181 MIPS Average
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.