[rbldnsd] Re: SIGSEGV in rbldnsd

[Chris Gabe]

On Oct 17, 2005, at 3:36 AM, Michael Tokarev wrote:

> [Cc'ing rbldnsd mailinglist.  Short summary of the issue:
>
>> One suggestion (maybe rbldnsd already does this), on the original   
>> malloc, allocate a bit extra.  This way a 100MB zone won't be  
>> wasted  forever when a few ip's are added to the zone file and  
>> realloc forces  a larger alloc.
>>
>
> Well, this is an interesting and quite old question - which
> "algorithm" to use for memory allocations for the large arrays
> of "unknown" size.  Standard technique is to double the size
> each time we hit the number of entries allocated and new entry
> is needed -- this is what's used in rbldnsd.  I also implemented
> some "optimisation", -- on next reload of the same dataset,
> rbldnsd remembers previous size of the array and allocates this
> amount of entries in first go, to reduce number of reallocations.
> And you're right, this techinique sometimes requires quite large
> amount of memory to be allocated and "wasted" during reloads.
>
> Two scenarios:
>
> "Classical" double-size-each-time, with, say, 4M+1 entries --
> rbldnsd will realloc(8M) to store the last entry, and right
> after that it'll realloc(4M+1) - so, during reload, it'll
> waste memory needed for (4M-1) entries.
>
> And with my "optimisation", the same happens when next reload
> will hit 4M+2 entries - rbldnsd will allocate the same 4M+1,
> and reallocate that to be (4M+1)*2, with similar realloc right
> after loading...
>
> Note this is contiguous memory of quite large size (for 4M
> entries on 32bit platform, we need (for ip4set) 4M*8 = 32
> megabytes of memory -- in comparision, dsbl.org main data
> is 7.3M entries now; for 64bit platform, the memory requiriment
> is twice of that).
>
> And I'm not sure how to "beautify" this algorithm.  Number of
> realloc()s *must* be "small enouth", because between two calls
> to realloc(), we allocate another chunks of memory to store
> other things (like txt templates), and the more reallocs we
> perform, the more fragmented memory will be, and the less
> chances to find large contiguous chunk of memory we will
> have.  So using "less aggressive" (compared to doubling
> memory size on each realloc - like, say, using constant
> increment each time) reallocation technique is as risky
> as this aggressive approach.
>
> Another way is to try to guess the final dataset size based
> on the size of files in the dataset and current "percent
> loaded".  Perhaps this is what I'm after, finally, but this
> will require some mods in many places of the code - to
> compute and pass that "current percentage" into loading
> functions and use it there.
>
> And for some quick fix, before I'll implement the above,
> I'll add some "rounding" to the "next allocation hint",
> so that instead of using 4M+1 next time, it'll use some
> larger value like 4M+(something more)...  Or to remember
> the last power of two used and start from there, not from
> really required amount of entries - which will be even
> simpler.
>
> Anyone have other ideas? ;-P
>
> Note this whole problem with "difficult" allocations only
> happens on freebsd so far... ;)

Allocation strategies, well, other than including some tailored  
memory allocator I have no suggestion.  But there may well be  
something out there that better suits rbldnsd's particular needs, and  
avoids per-O/S quirks in the stock malloc().

As for hitting memory limits:
Consider an option (not the default behaviour) to write the loading  
data to a file instead of allocating.  Then use mmap to map to it.   
This not only saves the space if you want to use the -f option (that  
is, you can free the original buffer before mapping the new one), but  
also allows the process to grow larger than with malloc.  At least on  
freebsd, we routinely go to 1.5GB using mmap, in spite of the per  
process 512MB limit on malloc.  dccd uses this mmap method; that's a  
dynamic database which grows periodically.  It has to worry about re- 
using that file on restart, which makes it more complex; rbldnsd  
wouldn't have that issue.

Another option for more memory would be to provide, at least on  
freebsd, a means to call setrlimit, so that one could configure a  
default per process memory size to a reasonable amount (to stop  
runaways), but a maximum of a larger amount.  So one's kernel config  
would have something like

     options     MAXDSIZ="(1024*1024*1024)"
     options     DFLDSIZ="(256*1024*1024)"

and rbldnsd would do this on startup, if invoked with an option that  
sets a variable called max_mem (a 64 bit value)

     getrlimit(RLIMIT_DATA, &r);
     r.rlim_cur = max_mem;
     setrlimit(RLIMIT_DATA, &r);

>>> For huge lists, it's better to specify separate dataset per
>>> "data source", instead of combining all sources in one large
>>> dataset -- even if you want to get only one zone.  This way,
>>> it's more maintainable, and requires less resources to (re)load
>>> data.  YMMV ofcourse ;)
>

As you pointed out there is a potential issue with exclude directives  
with the per data source method.  You wrote

> Instead of:
>
>  bl1:ip4set:l1,l2
>  bl2:ip4set:l2,l3
>  ...
>
> use:
>
>  bl1:ip4set:l1 bl1:ip4set:l2
>  bl2:ip4set:l2 bl2:ip4set:l3
>  ...
>
> *if* semantics(*) of this variant is ok with you.  This way,
> each of the l1, l2, l3 etc will be loaded only once.
>
> (*) the difference are "excluded entries", if any.  Well, it's
> possible to add "global" exclusions too, specifying additional
> file for every dataset, like
>
>  bl1:ip4set:ex,l1 bl1:ip4set:ex,l2
>  bl2:ip4set:ex,l2 bl2:ip4set:ex,l3
>  ...

Can you clarify what you mean by "difference"?  Do you just mean it  
creates two copies of the excludes?  Or that they don't work as you'd  
hope, i.e. if l1 has an exclude list then l2's trumps/erases that?   
If just the former it's not an issue in general probably; even the  
dul list from sorbs has only about 12% of the entries as excludes.

/cjg