[rbldnsd] TTLs and negative caching

[Michael Tokarev]

Jeff Chan wrote:
> On Tuesday, August 3, 2004, 4:16:30 AM, Michael Tokarev wrote:
> 
>>Jeff Chan wrote:
>>
>>>We run SURBL which has RBLs containing spam URI domains.
>>>They are used to block spam based on URI domains contained
>>>in message bodies.
>>>
>>>Currently some of our zone files have fairly long TTLs
>>>of several hours.  This appears to be causing new entries
>>>to take several hours to become active.
> 
>>Hmm....  That does not seem to be right.  What's your
>>zone and any test url/domain, for me to check?
> 
> 
> One zone with a long TTL is ob.surbl.org, with a record
> like ezherbalbuy.com.ob.surbl.org
> 
> with the SOA record:
> 
>   $SOA 21600 origin zone.surbl.org 1091531965 1200 600 604800 21600

Aha.  So, your negativeTTL is 6h, and so new entries will become
"active" (after NXDOMAIN query) after 6 hours, as caches will keep
the NXDOMAIN for exactly 6h.  (btw, in my opinion the form like
  $SOA 6h origin zone.surbl.org 1091531965 20m 10m 1w 6h
is much easier to read/understand.)

But looking at your zone/nameservers, I can't see how they're set
up currently, at least it's not obvious.  Namely, I don't see any
rbldnsd in between [a-hl].surbl.org, but e.g. [ijk].surbl.org
seems to be rbldnsd; different nameservers returns different results
(especially speaking of NS records), and some NSes seems to be
returning non-auth replies.  Looks like it'd be better to fix
that.. somehow ;)

[]
>>According to DNS specifications, negative TTL is taken from
>>the SOA record's minttl field (the last number in the $SOA
>>line).  So no, the TTL (either -t option or $TTL value) does
>>not affect negative TTL.
> 
> Perhaps we should be using a finer-grained approach to setting
> these.  Suggestions would be definitely appreciated. 

Hmm?  There are only 2 parameters to tune: positive TTL (for every
individual listed record) and negative TTL for all non-existing
records.  There's no ability to return different (smaller) negative
TTL for a record that is about to be listed.  The only option you
have is to adjust your negative TTL to find a sort of balance between
number of queries to your NSes and amount of time NXDOMAIN may be
cached (aka "false negative lifetime").  There's no other choice,
logically.

[]
> Long negative caches are ok if new entries (positive hits)
> can resolve quickly.  If we have a long negative TTL
> and a short positive TTL do new records become resolvable
> quickly?

By playing with positive TTLs, you're not affecting your
"false negative lifetime" (a time between entering an entry
into DB and when it becomes "active" after corresponding
NXDOMAIN expires from caches) in any way.  Only changing
negative TTL affects this time, and you're free to change
it while it does not hit your NSes very hard.  Just play
with negative (and positive as well) TTLs for a while
(set it to 3h for one day, set it to 1h for another day
and so on and watch your nameservers) - this way you will
figure out how exactly your NSes affects by the changes.
Usually, short TTLs causes very heavy DNS traffic, but
after some value, increasing TTL affects the DNS load
very slightly, like this:

load
   ^
   | +
   |  +
   |    +
   |        +
   |                  +
   |
   +-----=---------------> TTL
          \deadline, approx

> SBL has a very short minimum TTL of 5 minutes:
> 
>>sbl.spamhaus.org.       0S IN SOA       need.to.know.only. hostmaster.spamhaus.org. (
>>                                        2004080311      ; serial
>>                                        1H              ; refresh
>>                                        15M             ; retry
>>                                        1W              ; expiry
>>                                        5M )            ; minimum

I don't know why they're setting it this low, especially for
their data which is rather static.  For DSBL, minttl=10m is
ok, since it is trying to list new open proxies very quickly,
and having new entry within 10min after first abuse is quite
usual.  5min negative TTL for sbl seems to be too low.  But
anyway, all the TTLs only affects nameserver load (and amount
of queries every client does, ofcourse, but that's not our
problem, right? :)

/mjt