Author: Marc Bourzutschky
Date: 16:13:57 05/05/04
Go up one level in this thread
On May 05, 2004 at 18:32:36, Eugene Nalimov wrote: >On May 05, 2004 at 17:06:49, Marc Bourzutschky wrote: > >>On May 05, 2004 at 16:35:10, Robert Hyatt wrote: >> >>>On May 05, 2004 at 16:29:52, Marc Bourzutschky wrote: >>> >>>>On May 05, 2004 at 15:36:23, Robert Hyatt wrote: >>>> >>>>>On May 05, 2004 at 14:41:45, Marc Bourzutschky wrote: >>>>> >>>>>>On May 05, 2004 at 13:51:12, Robert Hyatt wrote: >>>>>> >>>>>>>On May 05, 2004 at 13:25:18, Marc Bourzutschky wrote: >>>>>>> >>>>>>>>On May 05, 2004 at 11:55:07, Robert Hyatt wrote: >>>>>>>> >>>>>>>>>On May 05, 2004 at 09:37:14, Marc Bourzutschky wrote: >>>>>>>>> >>>>>>>>>>On May 05, 2004 at 09:14:50, Mike Hood wrote: >>>>>>>>>> >>>>>>>>>>>On May 05, 2004 at 08:12:44, Vincent Diepeveen wrote: >>>>>>>>>>> >>>>>>>>>>>>On May 05, 2004 at 07:47:57, Mike Hood wrote: >>>>>>>>>>>> >>>>>>>>>>>>>I just let Filemon run while loading Fritz 8 to see why it takes so long. I was >>>>>>>>>>>>>shocked to see that during the initialisation Fritz tries to open every possible >>>>>>>>>>>>>tablebase. For instance... >>>>>>>>>>>>> >>>>>>>>>>>>>Open kpk.nbw.emd -- good, it's there >>>>>>>>>>>>>Open kpknbw.emd -- file not found >>>>>>>>>>>>>Open kpk_nbw.emd -- file not found >>>>>>>>>>>>>Open kpk_nbw_emd -- file not found (I never knew this format was valid) >>>>>>>>>>>>>Open kpk.nbw -- file not found >>>>>>>>>>>>> >>>>>>>>>>>>>And the same five accesses for the nbb file. >>>>>>>>>>>>> >>>>>>>>>>>>>Why carry on with the other three after finding the first tablebase? But it gets >>>>>>>>>>>>>even wilder when it comes to the 6-piece tablebases. All 365 possible tablebase >>>>>>>>>>>>>pairs in all possible formats are accessed, even though I don't have any on my >>>>>>>>>>>>>disk. Thousands of "file not found" results. Just one example, to show how >>>>>>>>>>>>>ludicrous it is: >>>>>>>>>>>>> >>>>>>>>>>>>>First Fritz tries to open krbnkp.nbw.emd, krbnkpnbw.emd, krbnkp_nbw.emd and >>>>>>>>>>>>>krbnkp.nbw.emd. Almost the same as before, except Fritz is assuming 6-piece >>>>>>>>>>>>>tablebases are compressed. But then Fritz tries to open krbnkp.0.nbw.emd, >>>>>>>>>>>>>krbnkp.0_nbw.emd, krbnkp.0nbw.emd and krbnkp.0_nbw_emd. Then krbnkp.1.nbw.emd, >>>>>>>>>>>>>etc... and krbnkp.2.nbw.emd... and all the way through to krbnkp.g.nbw.emd. That >>>>>>>>>>>>>means 136 disk accesses for a tablebase that I don't have! And that's only one >>>>>>>>>>>>>tablebase out of 365. >>>>>>>>>>>>> >>>>>>>>>>>>>Wouldn't it be much easier just to scan the tablebase directory and only open >>>>>>>>>>>>>the files that actually exist? >>>>>>>>>>>> >>>>>>>>>>>>Both nalimov and i do this in a similar way. >>>>>>>>>>>> >>>>>>>>>>>>If you are willing to write code for this that works faster and works both for >>>>>>>>>>>>windows and *nix, then i will be real happy to use it. >>>>>>>>>>>> >>>>>>>>>>>>Best Regards, >>>>>>>>>>>>Vincent >>>>>>>>>>> >>>>>>>>>>>Thanks for the info, Vincent. I assumed the initialization code had been written >>>>>>>>>>>by Chessbase, not by Eugene. >>>>>>>>>>> >>>>>>>>>>>My math was a bit off in my original post, but after looking at Filemon's log I >>>>>>>>>>>can give the exact figure: Fritz attempts to access 33647 non-existent tablebase >>>>>>>>>>>files. And please... you can't tell me that if the file krbnkp.0.nbw.emd doesn't >>>>>>>>>>>exist it still makes sense to look for krbnkp.1.nbw.emd, krbnkp.2.nbw.emd, etc >>>>>>>>>>>all the way to krbnkp.g.nbw.emd. That's a waste of processor time on any >>>>>>>>>>>operating system. >>>>>>>>>> >>>>>>>>>>As this is only done once during initialization it is not such a big deal. IMHO >>>>>>>>>>a more serious nuisance is that all available endgames on the paths are >>>>>>>>>>initialized even though they may never be used. As a fair amount of memory is >>>>>>>>>>taken up by each endgame that is initialized this is a serious inefficiency. >>>>>>>>>>I'm surprised that Fritz and Co. have not implemented a scheme where an endgame >>>>>>>>>>is only initialized when it is actually required. >>>>>>>>>> >>>>>>>>>>-Marc >>>>>>>>> >>>>>>>>> >>>>>>>>>Would you _really_ want to wait until you have a few seconds left, with no >>>>>>>>>increment, then start opening files, malloc()'ing buffers, setting up the >>>>>>>>>decompression stuff? Oops. flag just fell. >>>>>>>>> >>>>>>>>>:) >>>>>>>> >>>>>>>>I actually do just that and it takes a small fraction of a second to initialize >>>>>>>>one tablebase. For me at least the likelyhood of this being an issue is >>>>>>>>miniscule compared to the amount of memory I can save. >>>>>>> >>>>>>> >>>>>>>What will you use it for? And if memory is tight, that "fraction of a second" >>>>>>>will grow as you might have to page out an inactive process to make room... >>>>>> >>>>>>For one, I can use the extra memory for hash tables and other things I might be >>>>>>doing on my computer. >>>>> >>>>>We are not on the same page apparently. >>>>> >>>>>Your idea won't work. Because inside the search, when you _do_ need to access a >>>>>table, you need to malloc() memory for the decompression indices and read them >>>>>in. That takes time. If you used all of memory for hashing you just introduced >>>>>significant paging overhead that is going to slow you further. If you want to >>>>>dynamically reduce the size of the hash table inside the search, forget about >>>>>it. That adds so much overhead it isn't thinkable... >>>>> >>>> >>>>I already have it working! Of course, if you allocate all free memory to hash >>>>tables initially you might run into the problem you describe. But even that can >>>>be easily fixed by maintaining an index cache, just like there is already a >>>>cache for the tablebase values themselves. >>> >>> >>>Either you use more memory for hash, as you said you would, and run into paging >>>when you start probing tables, or you don't use more memory for hash, which >>>means you "reserve" the necessary EGTB memory but don't use it until needed. >>>When you do need it, you will lose games on time if they are short time >>>controls. >>> >> >>The point is that for practical purpose the number of tablebases actually needed >>is way less than the total number possible. >> >>>One way or another, the current approach is the correct one. Why do you think >>>_everybody_ is doing it that way??? >> >>First of all, the current approach is not incorrect, just memory inefficient. >>Not a big deal if you only use 5-man tables, but a bit more of a deal once all >>the 6-man tables are there. Also not a big deal if you have lots of memory to >>burn. Second of all, _everybody_ is just Eugene Nalimov himself, since people >>just copy his code. I'm sure Eugene would agree that with a suitable index >>cache one can eliminate loading all the tablebase stuff on startup, with a >>miniscule chance of this leading to losing games on time. > >Yes, that can be done. Something like: >* Reserve some memory not only for TB hash, but also for indices. Do not >pre-load indices. >* When you want to read from TB whicj indices are not in memory, read indices >first. If You run out of memory for indices, unload some index. > >It will somewhat slow you during probing, but I doubt you'll lost games because >of this. After all you are reading one extra contigious block from disk. > >I did not implement it because I never thought that will be a problem. Right now >that is a problem only for *some* CC entusiasts, who have slow disks and lot of >TBs. 99% of users probably are not noticing those several seconds. > >BTW, current code (in Crafty, and in newer ChessBase programs) handle >initialization better than old one. I got reports that users often set TBPATH >(or whatever it is called) pointing to the CD-ROM, and modified the code to >better handle missing disk in CD-reader. > >I would see what I can do to improve initialization time now, but I will not >implement Marc's suggestion. Too much work, and I'd better work on something >else. > >Thanks, >Eugene I did not suggest you should implement this, but I'm a little surprised the ChessBase guys have yet not done it. The issue ist not the few extra seconds at startup, but the huge memory requirement at run time. Already with the handful of 6-man tablebases available on DVD it taxes some systems, so ChessBase recommends not loading all of them on initialization. The index cache would eliminate this issue altogether. Of course, if you have gobs of memory this it not an issue. For me personally, since I'm not interested in 1 0 bullet games, I don't use an index cache at all, but simply run the initialization on demand for any specific endgame. This took almost no coding and I have not had any problems with it at all. -Marc
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.