[rbldnsd] Memory allocation questions

[Amos Jeffries]

Michael Tokarev wrote:
> Scott Knight wrote:
> 
>> Hello,
>>
>> I am trying to get a better understanding of how
>> rbldnsd allocates memory – both on initial startup and
>> during a reload.
> 
> 
> The allocation "algorithm" is very simple and isn't
> much different for initial load of data and subsequent
> reloads.
> 
> There are two different "classes" of memory being
> allocated for ip4set: one is a "TOC", large arrays
> to hold the IP addresses (4 bytes) and pointers to
> the resulting A+TXT strings (another 4 bytes on 32bit
> platform); and another large "class" is the strings,
> many small objects.
> 
> When loading the data, rbldnsd can't know how much
> entries it will need for the large arrays.  So it
> starts with some number of entries (say, 16), and
> grows the array (realloc) multiplying current count
> to 2 each time it needs more entries.  So, for example,
> for 4M entries it will allocate array of size 4M*8
> (or *16 for 64bits), and for 4M+1 it will be 8M.
> 
> When the whole file is read, rbldnsd does final realloc,
> reclaiming extra memory.
> 

<snip discussion of present allocation method and fragmentation.>

> 
> Well..  I thought we may run into this problem sooner or later,
> and here we go, it seems.
> 
> I *guess* you aren't running linux (nothing wrong with that
> really ;) -- because on linux and with default malloc(),
> things don't work the way as per above.  Instead, for
> larger objects, malloc() uses mmap(), which in turn uses
> powers of virtual memory to avoid the above fragmentation
> entirely.
> 

I am thinking we have reached a point where it becomes necessary to 
check the file size before allocating memory. If you know how much data 
is being loaded you can estimate a close figure to the final memory usage.

The algorithm could then be: open file, check file length, allocate n 
bytes, on array exhaustion allocate n+n/2 bytes (or similar).
slightly tricky extra location of file length, but much reduced memory 
usage for large zones.

-- 
Amos Jeffries
--
www.treenetnz.com
--
We are fast approaching the time when "packets from a M$ operating 
system" is synonymous with "hostile behavior".