[rbldnsd] [RFD] [long[ish]] to be or not to be...

[Michael Tokarev]

This is a somewhat long request for comments/discussion.

After seeing many quite different feature requests, I
come to an idea.  Which is as follows.

Currently, rbldnsd allows to have a set of datasets per
zone, and results from all of them are united (ORed)
together to form a result for particular request.  For
example, a zone may consists of a list of dialups (ip4set),
a list of proxies (ip4tset) and a generic dataset which
holds some metadata.  When queried, the result will be
a union (OR) from all 3.  So far so good.

Long time ago there was a request for a "global whitelist"
(hello Matthew!).  The idea was to have a way to whitelist
our own relays as to not lose our own internal email.  This
can only be done by adding our ranges into each individual
dataset, which is problematic for ip4tset for example (no
exclusions there), and troublesome in general (have to add
it to EVERY zone).  This -- global whitelist -- isn't
implemented still.

Next come ACLs.  A way to modify result depending on WHO
the client is.  With various fancy stuff.  It's actually
working now.

Later on, when data sets become large, there was a need
to quickly add exclusions without going full rsync+reload
cycle which is long and resource-hungry.  Somewhat similar
to "global whitelis" above, and very similar to what is
done now for dsbl.org - their "fresh" pieces for the main
data, which gets updated frequently, are small, fast to
propagate and reload, and which gets merged into main data
from time to time -- this allows, even when the main set
contains >13M records and reaches almost 200Mb in size --
to propagate new additions to all secondaries in 1-2 minutes!.
Dsbl.org variant works since the two datasets (main large
and fresh small) are united together nicely.  It doesn't
work for exclusions, since exclusions are local for a
dataset, so full reload is still necessary (even if the
exclusions will be replicated in a small file).

Next there was a rather interesting request, to have
several (many) subzones each containing the same data
and is accomplished with its own small ACL - so that
a given subzone can only be queried from a given location.
The key point here is that the data itself is exactly
the same for every subzone (once ACL is Ok), and there
are many such subzones.


All the above can be solved in a rather clean and elegant
way (IMHO anyway), once and for all.  By changing current
list (union) of single elements to union of CHAINs, where
elements in a chain is ANDed together.  For example, an
ACL becomes:

  bl.ex.com:acl:acl-data-file:ip4set:bl-data-file

It's a zone name, followed by a SET of (type:file) PAIRS
(yeah, each being a dataset!), instead of current SINGLE
(type:file) element.  Here, once one dataset has some
answer (be it "listed" or "clean" - think whitelist),
we stop processing of this chain and go to the next one.
(yes, the whole specification becomes quite long and
quite unreadable, but see below).

Another example with "quick exclusions":

  bl.ex.com:ip4set:exclusions:ip4set:main-data

where file "exclusions" contains something like:

  !127.0.0.0/8
  ...

when the query is for 127.0.0.1, first dataset has the
answer, which is "not listed", so we don't continue to
the "main-data" dataset but return NXDOMAIN right away,
even if "main-data" may have something for that IP.

This way, ACLs and such exclusions may be freely combined
with each other, so there may be more than one ACL for
example, each zone may have "global whitelist" applied to
it and so on.

Here, more exotic example with subdomains-with-ACLs.  The
same principle:

  bl.ex.com:subdom-acl:sdacl:ip4set:main-data

Here, an (non-existing) "subdom-acl" dataset type handles
file ("sdacl") of the form:

  subdom1  127.0.0.0/8 10.0.0.0/8
  subdom2  192.168.0.0/16
  :listed:don't mess with our data, it's not yours

Ie, queries like 2.0.0.127.subdom1.bl.ex.com, but only if
coming from loopback or 10.* range, are processed by the
"main-data" dataset, and the arqument will be 2.0.0.127.
Similarly, 3.0.0.127.subdom2.bl.ex.com from 192.168.*
are processed by the same main-data as 3.0.0.127.  For
the rest, we always return "listed" with the mentioned
TXT record.   Here, our dataset not only checks something
(here it checks bot the query and client's IP), but also
MODIFIES the query somewhat, cutting a subdomain off it.


Ok, good stuff (or not... ;)  It's time for some salt.

Rbldnsd is a glue between DNS and some other things like
set of IP addresses.  And this glue turned to be pretty..
ugly, so to say.  Because some things aren't that easy.

For example, I know at least one place where rbldnsd
violates DNS standards pretty much blatantly.  Consider
a query like 2.0.0.127.bl.example.com, when 127.0.0.2
is really listed - it will be returned as existing
domain, with A and TXT records and so on.  Now think
about query 0.0.127.bl.example.com -- this query will
be answered with code NXDOMAIN.  But how come, child
is here but no parent?  It can't be!  A recursive
resolver, in theory, when asked for 2.0.0.127.bl.ex.com,
can EXPLICITLY try each of com, ex.com, bl.ex.com,
127.bl.ex.com etc in turn to find the final answer -
there's nothing in DNS specs preventing such a
behavour.  It's just impractical to do so - instead,
it asks each of corresponding nameservers the whole
name, not components.  But if some NS did that, it
will stop on 127.bl.ex.com and will return NXDOMAIN
to the original query - because rbldnsd will return
NXDOMAIN for the 127 query!  Sure it is possible to
accurately find that 127 actually is "listed" as first
octet of 127.0.0.2, but that requires more code and
slower operations...  So we live with this blatant
violation of the standard - as a trade-off between
practical usage, robustness and standards.

Ok, back to the original explanation.

As a glue between something and DNS world, rbldnsd
has to deal with "DNS metadata" - things like NS and
SOA records, and also various "sugar" things like
TXT records for the whole zone explaining what it
is, (probably fake) MX records etc.  The most interesting
thing here are NS records, and especially "glue records" -
A records corresponding to the NS records.

Suppose we're running bl.ex.com zone, which has nameserver
ns.bl.ex.com - yes, within the zone.  With address (A
record) 10.0.0.1.  And suppose we have an ACL for our
zone, which returns "listed" (127.0.0.2) for EVERY query
made by certain client.  The question is -- what to reply
to that said client if it asks for A for ns.bl.ex.com ?
By all means we should return 10.0.0.1, not 127.0.0.2!
But how "ns" is special compared to, say, "1.0.0.127" or
"spammer.com" (no, it's not as simple as "single label"
vs "multiple labels" :)

Currently, this problem is solved with a rather ugly
hack.  We allow only single type of dataset - namely,
the "generic" dataset - to answer to queries even if
ACLed out like that.  Ie, we run the query over ACL,
remember the result, next probe all the other datasets,
ignoring everything BUT generic.  At the end, if we
have something, we reply that; if not, we add our
"everyone listed" stuff instead.  So the end result
will be that for ACLed out clients, they will see
everything listed in generic dataset, and for the
rest the result will be "listed".  Not a good solution,
but at least more or less practical.  Hackish code.

Now let's change the interface to a "new modern way",
using list of "chains".  I can drop the hacks from
the code (both special handling of ACLs and special
handling of generic dataset) and put generic dataset
IN FRONT of an ACL to achieve the same effect without
hacks.  Clean nice maintainable code.

But now new question arises.  How to combine several
chains together, provided some have ACLs returning
"everyone listed".  E.g.

  bl.ex.com:ip4set:dialups \
  bl.ex.com:acl:bl-acl:ip4set:proxies \
  bl.ex.com:generic:metadata

This is a single zone with 3 chains, ACL applies to
one chain only.  In practice it's probably not very
useful, but i still can think of -- like, "don't show
our proxies to anyone".  So, we added "everything
listed" to the reply when processing 2nd chain, and
finally has to add A record for our ns.bl.ex.com which
is in the original query... The result will be
127.0.0.2 "listed" and our 10.0.0.1, which is even
more wrong.  Maybe an operator error after all --
either list the same ACL for every chain, or don't
use "listed" ACL verdicts.  But the former adds
more to the general command-line ugliness.

There are some other similar "corner cases" exists.
All comes from 3 features which I dislike in their
current form, namely:

  a) glue records.  Initially, rbldnsd REQUIRED
    a real nameserver, if not only to keep A records
    for the NS records.
  b) ACLs, especially ACLs with "everything listed"
    and "everything not listed" verdict.
  c) combined dataset.  With new way of things, this
    becomes rather odd, "orphan", so to say - it already
    is somewhat not in line, due to reload of whole thing
    and due to wrong working of NS/SOA stuff, and it
    becomes even more out of line.

But c'est la vie.

Aha, config ugliness.

Since config file for rbldnsd is a shell fragment, it's
easy to reduce the config to something more readable,
by defining and using shell variables, like this:

dsbl_list=ip4tset:dsbl/rbldns-list.dsbl.org
dsbl_flist=ip4tset:dsbl/rbldns-fresh.list.dsbl.org
gacl=acl:global-acl
dsbl_acl=$gacl:acl:dsbl/acl

RBLDNSD=...
  list.dsbl.org:$dsbl_acl:$dsbl_list \
  list.dsbl.org:$dsbl_acl:$dsbl_flist \
  ...

Here, we define common short variables and use them
later when defining actual zones/datasets.  Note how
dsbl_acl is defined - it includes gacl (global acl).
The above is a new equivalent of current:

  acl:global-acl \
  list.dsbl.org:acl:dsbl/acl \
  list.dsbl.org:ip4tset:dsbl/rbldns-list.dsbl.org \
  list.dsbl.org:ip4tset:dsbl/fresh-rbldns-list.dsbl.org

ie, a global acl, one acl for list.dsbl.org zone, and
two datasets of type ip4tset.


Ok.  So here are the questions, finally... ;)

  o to start with, if this whole stuff will be implemented,
    next version will be rather incompatible, and it'll be
    the first really incompatible version.  I don't like
    this at all, but from another point of view this way
    eliminates quite some ugliness and makes things more
    strightforward and clean.

  o while at it, maybe remove glue records and combined
    dataset entirely? :)

  o there are other ways exists to implement SOME of the
    features.  For example, to make the chains to be a
    property of certain dataset types, not zone property -
    i.e., "acl" dataset will allow to "connect" some other
    dataset to it, but not any dataset type will allow this.
    That way, not every dataset will be "combinable",
    ugliness will increase (in both code and documentation
    and understanding of things) and flexibility will
    decrease.

  o when we collecting information about a particular zone
    (after a (re)load), we cache A records for our NS (glue
    records, that is).  For this, we run our NS records over
    all our datasets as if it were some client asking for
    A ns.bl.ex.com..   But now, if an ACL is on the way,
    what we should do?  (Currently an ACL is ignored here;
    the difference from regular query is that there's no
    client per se).

  o what are our choices for situations like outlined,
    e.g. when client asks for A records and ACL is present.

Or.. something like that...

Comments, anyone?

Thank you for your attention.  And please excuse me for
this long email.  I'm quite stuck now, because I don't
know where to go from here.  One half of me tells me
"lets broke it all for a better world", while another
half says "hey, think about poor users who will have
even no way to keep their config compatible!".  Oh well...

/mjt