[rbldnsd] Problem...

[Michael Tokarev]

Matthew Sullivan wrote:
> Hey all,
> 
> Someone just posted an interesting issue with the SORBS DNSbl....  This 
> probably documented behaviour, but I figured I'd pass it by the list and 
> see if anyone including Michael has any comments...

Yes I have, and in fact I told you about this several time since
the beginning.

> We have supplied the zones as just listings until recently when we 
> picked up the Dynablock and started actively maintaining it in the place 
> of Ben.  However the issue that appeared today -
> 
> We have 17740 exceptions listed, and these exceptions are exported into 
> the DUHL where all works fine......except the DUHL zone is aggregated 
> into the main zone, so all the exceptions are also aggregated into the 
> main zone which means anyone excepted from the DUHL is actually excepted 
> from all listings when using the aggregate zone....

Sure.  There are 2 different ways to combine several different
"data sets" into one.  First is to create a single rbldnsd dataset
for all them, like:

  dnsbl.sorbs.net:ip4set:dynablock,spam,http,socks,...

and second is to use several datasets for the same zone:

  dnsbl.sorbs.net:ip4set:dynablock \
  dnsbl.sorbs.net:ip4set:spam \
  dnsbl.sorbs.net:ip4set:http \
  dnsbl.sorbs.net:ip4set:socks \
  ...

Exceptions are local for the dataset, regardless of how
you build the dataset.  So you're able to specify e.g:

  bl.example.com:ip4set:spam,spam-exceptions

thus maintaining list of spam sources and exceptions for
that in separate files.  But they don't span "dataset
boundaries".  That is, in second form (several datasets
for the same zone), you will get what is needed: DUHL
exceptions (listed inside the dynablock file) will be
applied to the DUHL ONLY, without affecting the rest
of the data.  More, this way, each dataset will be
reused for both corresponding subzone (any number of
them in fact) and for the combined zone, consuming no
additional memory.

> Now I figure I have a few options...
> 
> 1/ Process out all the exceptions and stop using them
> 2/ Process out the exceptions at export time and create an extra zone 
> which is the aggregate zone
> 3/ Talk to this list and Michael about whether the exceptions can be 
> limited to apply to their own zone only.  (Guess where I am ;-))

Yet again.  Rbldnsd tries to keep away notion of the "zone",
since main data for a DNSBL is a list (set) of IP addresses,
not something DNS-related.  Try to look at it from this point
of view.  You give it a list of data sets, and specify how they
will be "mapped" into zones.  Build the datasets the way you
want them to be, and think in terms of datasets.  You have
different datasets - DUHL, spam list, proxies etc.  Not subzones,
but datasets.  When you build those, you map them into zones,
where your main (combined) zone consists of SEVERAL DIFFERENT
DATASETS.  It is not the "combined dataset", it is a combined
ZONE, which consists of several different datasets (logically,
based on the nature of the data, not where the data will appear
in DNS).

In order to optimize stuff WHEN THE DATASETS AND MAPPING TO ZONES
will work ok, rearranging the datasets may help somehow (because
several datasets per zone means several lookups for rbldnsd, while
one dataset with all the data, while requires more memory, means
only one lookup in larger table).  Original (logical) datasets
may be freely combined if they don't have exceptions which you
only want to be applied to some sets and not others (eg DUHL
exceptions should not be applied to list of proxies and relays,
but neither proxies nor relays have exceptions so both may
be combined with each other).  But I don't suggest you to try
to optimize things here, because it is much simpler to maintain
and think about the whole stuff in terms of your data sets,
not in terms of zones.

> Comments/suggestions welcomed.
> 
> Example Data:
> 
> $DATASET ip4set http @
> $SOA    172800  rbldns0.sorbs.net. dns.isux.com. 1077267901 7200 7200 
> 604800 3600
> $NS 48h rbldns0.sorbs.net.
> :127.0.0.2:HTTP Proxy See: http://www.dnsbl.sorbs.net/cgi-bin/lookup?IP=$
> $TTL 172800
> 4.3.216.194
> 4.17.224.68
> 4.21.138.56
> 4.22.136.147
> $DATASET ip4set dul @
> $SOA    172800  rbldns0.sorbs.net. dns.isux.com. 1077267901 7200 7200 
> 604800 3600
> $NS 48h rbldns0.sorbs.net.
> :127.0.0.10:Dynamic IP Address See: 
> http://www.dnsbl.sorbs.net/cgi-bin/lookup?IP=$
> $TTL 172800
> 4.2.0.0/15
> 4.10.0.0/15
> 4.12.0.0/14
> !4.3.216.0/24
> !4.3.218.0/24
> 
> .....
> 
> Using the above data you get the following:
> 
> $ host 194.216.3.4.dul.dnsbl.sorbs.net
> Host 194.216.3.4.dul.dnsbl.sorbs.net not found: 3(NXDOMAIN)
> 
> $ host 194.216.3.4.http.dnsbl.sorbs.net
> 194.216.3.4.http.dnsbl.sorbs.net has address 127.0.0.2
> 
> All fine and good... now the problem:
> 
> $ host 194.216.3.4.dnsbl.sorbs.net
> Host 194.216.3.4.dnsbl.sorbs.net not found: 3(NXDOMAIN)

Hmm.  And this is something different.  You're using
several DATASETS for the combined zone here, not one
large dataset, so any exception is local for the dataset,
not for the zone.  Lemme see... hmm, now, 4.3.216.0/24
is NOT listed as an exception.

> I expect and don't get (my fault I presume):
> 
> $ host 194.216.3.4.dnsbl.sorbs.net
> 194.216.3.4.dnsbl.sorbs.net has address 127.0.0.2
> 
> Removing the DUHL exception (!4.3.216.0/24) I get:
> 
> $ host 194.216.3.4.dnsbl.sorbs.net
> 194.216.3.4.dnsbl.sorbs.net has address 127.0.0.10
> 194.216.3.4.dnsbl.sorbs.net has address 127.0.0.2
> 
> which is expected.

It should Just Work (tm).

> Suggestions, comments?

Take a look how rbl[12].oregonstate.edu
nameservers are set up.  If they're using
single dataset for combined zone instead of
several smaller datasets as the rest of your
NSes does, that's what you will get when
querying one of them.  It's difficult to
say whenever I'm right or not here, because
of the lack of test data.

/mjt