[rbldnsd] memory usage 32bit vs 64bit

[Michael Tokarev]

Per Jessen wrote:
> I've just noticed a difference in memory usage depending on the type of
> system - I have the exact same rbldnsd config running on a 32bit system
> and on a 64bit system. 
> 
> On the 32bit system rbldnsd uses up 408M (according to top)
> On the 64bit system rbldnsd uses up 649M (according to top)
> 
> I was just wondering if this is unintentional or fully expected?

It's not "fully" expected, but still not unexpected either.

Yes pointers are twice as long on 64bits compared to 32bits.
But the final memory usage depends on the dataset.

For example, ip4set uses a pointer for each entry - hence it
will use 2x the size for pointers (*).  Note that a pointer
isn't really necessary for each entry, as long as you use the
same text for each -- in this case each pointer points to the
same memory location(**)...

But ip4Tset uses the same amount of memory regardless - 32bits
per entry.  Or at least should (if not it's a bug).  Because
it only stores an IP address per entry, all entries sharing
the same common A+TXT template.

(*) things are a bit more complicated.  Each entry in ip4set
is like this:
  struct entry {
    uint32_t ipaddr;
    char *rr;
  };
On a 64bits platform, that is still 2x64bit (16 bytes), even if
real space needed is 8+4 = 12bytes only - due to padding/alignment
issues (**).  In contrast, ip4tset is like this:
  struct entry {
    uint32_t ipaddr;
  };
which aligns on 4 byte boundary even on 64bits arch.  Or should.

(**) In theory I can optimize memory usage here, by using an
indirect reference to A+TXT template - building a table (an
array of pointers) to A+TXT things and using an index into
this table in struct entry.  This will not slow down things
and will only require 8 bytes per entry even on 64bit arches.
It saves nothing on 32bits, and when I wrote rbldnsd 64bits
arches were not this common as nowadays.  Also, such an indirect
scheme is necessary for on-disk format I'm still planning to
implement for rbldnsd-1.0.  So I think one day 64bit memory
usage will shrink back to one needed for 32bit, and it should
be soon... ;)

By the way, 400mb is quite a few -- looks like you
loaded alot of data... ;)

Thanks.

/mjt