Author: Edward Screven
Date: 14:03:01 06/19/98
Go up one level in this thread
On June 19, 1998 at 00:30:13, Robert Hyatt wrote:
>>got it. the next step you might consider is eliminating the first
>>intermediate file. instead of streaming the unordered records to
>>a file, write them directly into the memory sort buffer. each time
>>the buffer fills, sort it and spill it to disk as a run. merge and
>>aggregate as before. that should cut the disk space required in half.
>>
>>if you want to try the key range partioning trick, you would apply
>>it at the point you write the records into the memory buffer -- just
>>discard the records that aren't part of the current pass.
>>
>> - edward
>
>
>it actually doesn't change a thing, because I first have to sort/write,
>which takes O(N) space, then I have to merge, and during the merge, I
>need O(2N) space.
your scheme as i understand it:
1. you create a file containing unorder records of size N
2. you read in the file and create sort runs of total size N
3. you remove (i'm guessing) the file in step 1
4. you merge the sort runs, create a file of size M
the disk space you require (in addition to the pgn) is
max(2*N,N+M)
my understanding is that you use a minimum number of times
played to reduce the entries in the resulting book. i assumed
that you applied this at merge time (since its trivial to do so.)
setting "minplay" to just three should make M << N. so by
skipping step 1 you could go from requiring 2*N to a little
more than N disk space.
this is more or less how i create my book. my records are
different than yours because i include moves. they look like
<position,move,resultinfo,flags>
so i sort by position and move. my intermediate record format
is 16 bytes long and i require around 400M of temporary disk space,
in addition to about 6M of space for the final result when processing
wall.pgn.
- edward
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.