[rbldnsd] TTLs and negative caching

[Michael Tokarev]

Jeff Chan wrote:
> On Tuesday, August 3, 2004, 4:51:56 AM, Jon Lewis wrote:
> 
>>set a small TTL via the SOA record (this
>>will be used as the negative cache TTL).  You can set the positive answer
>>TTL separately via a -t commandline arg or $TTL special in datasets.
> 
> 
>>If you look at the NJABL servers, positive answers have a TTL of 21600,
>>but negatives have 900.
> 
> 
> That makes some sense, as if to say "remember the positive
> entries longer than the negative ones."  But it seems to
> differ from Michael's intuition that there is a lot more
> negative caching than positive, so a short TTL on negative
> would have a much larger effect on increasing traffic.

It isn't intuition.  Compare
   http://spambusters.org.ar/nsg/m_q.gif - total queries, with
   http://spambusters.org.ar/nsg/m_p.gif - positive replies
e.g. for list.dsbl.org, there are 712 queries/sec total and
208 positive replies/sec, which means that more than 1/3 of
all queries gets positive replies, even when positive TTL is
about 35min while negative TTL is only 10min.  That to say --
there are more "bad guys" sending mail (trojans/proxies on
adsl networks) than real mailservers.  With that numbers,
both negative and positive TTLs affects NS load almost at
the same level.  But with domains, I expect hit ratio to be
much lower (many trojaned machines compared to number of
legitimate mailservers, but only a few spammy domains compared
to all domains seen in URLs) -- in this situation, negative
TTL will affect nameservers more significantly than positive
TTL.

> The thing to keep in mind is that most of the queries to
> an RBL result in negative responses, as in "this domain
> or ip is not on the list" since the lists are much smaller
> than the superset of everything else on the Internet, which
> might be checked against an RBL.

Well, not really (see above) :)

When tuning the TTLs, just choose numbers (by "try and see"
method which is easiest to perform) that will generate
acceptable load on nameservers while still having acceptable
"response time" (false positives and negatives).  For a
relatively static data (like SBL for example), a TTL of several
hours (1..3, maybe even 6) seems to be acceptable.  I don't
know how it applies to surbl data - that is, how much spam
a spammer able to send using new domain while it will be
expiring from ob.surbl.org caches.

/mjt