[rbldnsd] prerelease: rbldnsd-0.996.98, introducing zlib support

Michael Tokarev mjt at tls.msk.ru
Thu Dec 22 01:10:31 MSK 2005


As per previous RFC, I implemented transparent on-the-fly
decompression support for gzip-compressed data.  Rbldnsd,
when compiled with zlib support, is now able to read
gzip-compressed files as if they weren't compressed, ie,
instead of

   rbldnsd ... zone:ip4set:data ...

one now able to specify

   rbldnsd ... zone:ip4set:data.gz ...

(assuming `data.gz' is gzip-compressed `data').

Decompression is turned on when rbldnsd sees gzip magic
in a few (really 2) first bytes of the file (NOT by
file extension of anything else).  At the same time,
when new rbldnsd (without compiled-in support for this
feature, which is optional) detects the data is compressed,
it aborts loading it.

This feature may be turned off by a command-line option, -C
(capital C), just in case...

While doing this stuff I was forced to write a small
"stdio replacement" where one is able to plug the
decompressor (zlib) - gzio in zlib is VERY SLOW, so I
attempted to write my own, which resulted in this 'istream'
module, which is a light stackable buffered-reading layer.
Using it alone, without zlib etc, resulted in a noticeable
(re)load speed improvement compared with glibc - on my
machine, loading dsbl data (7.8M lines, over 100MB size)
using stdio takes ~17sec, while with this new istream,
it only takes 8 sec -- two times faster, which is very
surprizing especially having in mind that - in theory -
it should spend significant time when sorting etc...
And when reading the same data compressed (which is 21MB
in size), it takes about 13 sec.

So, take it as an added bonus -- even if zlib support will
be proven wrong, (re)load speed improvement is here.. ;)

There are some other changes as well, including rewritten
configure script which now accepts some --enable-* and
--disable-* options.

Please note this is NOT a production release, It Contains
Bugs (tm) which I don't yet know about.  Only for testing
and review.

Also please note that this zlib stuff is somewhat useless
until it's in production for quite some time so that DNSBL
maintainers will start publishing .gz files (if there're
other advantages ofcourse).

The PRErelease is available at
   http://www.corpit.ru/mjt/rbldnsd/pre/rbldnsd-0.995.98.tar.gz

Enjoy.

/mjt


More information about the rbldnsd mailing list