[rbldnsd] out of memory error

[Michael Tokarev]

14.10.2013 19:01, Dave Lugo wrote:
> On Sat, 12 Oct 2013, Dave Lugo wrote:
>>
>> Hi,
>>
>> I'm trying to load a very large zone into rbldnsd.
>>
>> It's a dnset comprised of 8 files, ranging in size from a few MB to almost 2GB.  Total size of files for this dnset is about 2.5GB.

That isn't very large, I don't think.  Provided your binary
is 64bits ofcourse - this is obviously over the top for 32bit
userspace.  Other than that it should work.  Is it 64bits?

But I really question this usage.  While for storing IP addresses
rbldnsd is good, maybe it isn't that good for arbitrary domains.
I never tried to optimize it much in this area.

With this amount of data, "reload/query" ratio becomes quite bad.
I mean, amount of time/resources spent for (re)loading the data
is large compared with the time/resources used when actually
answering queries.

It looks like you're better off using some on-disk database to
back up a nameserver.  Maybe pdns with an sql backend, or something
like that.  This isn't nice to replicate to secondaries when you
don't run them, but in some cases it should do the trick.

> It's running RHEL 6.3, 64 bit kernel.
>
> ulimit* showed a stack size of 10240KB, I increased it to 40960KB, but
> am still seeing the out of memory error.

No, rbldnsd does not need much stack size.  It uses malloc() alot,
and tries to allocate larger chunks.

It all smells pretty like a 32bit binary with 32bit address space.


I just did a quick test.
I created a file with (arbitrary) 52268396 lines containing
single (unique) word each of 61 byte long (generated using
perl one-liner).  Total size of this file is 3240640513 bytes.

rbldnsd loaded it fine (but a bit slow):

~/src/rbldnsd/rbldnsd -n -b 127.0.0.1/1053 foo:dnset:data
rbldnsd: listening on 127.0.0.1/1053
rbldnsd: dnset:data: 20131014 153959: e/w=52268396/0
rbldnsd: zones reloaded, time 42.41e/33.85u sec, mem arena=-923228 free=178 mmap=816696 Kb
rbldnsd: rbldnsd version 0.997a (23 Jul 2013) started (1 socket(s), 1 zone(s))

Memory reported by top(1) on linux is 3.9Gb (which is ofcourse
larger than the size of the file, due to simplicity of the
data structure).

It looks like the memory map report is wrong (32 vs 64 bit number
is used somewhere), but other than the report itself, it all works.

Thanks,

/mjt