[rbldnsd] RFC: Data expire support

[Amos Jeffries]

Comments after each point.

----- Original Message ----- 
From: "Michael Tokarev" <mjt at tls.msk.ru>


<snip people complaining>
>
> So, after several small discussions here and there, we come across
> this simple idea: a data file may contain an 'expire marker', ie,
> a timestamp indicating when the data becomes invalid.
>
> Yes there is `expire' field in the SOA record, which is not used
> currently, but I'm for another way to specify this expire time,
> because of several reasons:
>
<snip good points, I agree with all>

Additionally. With a seperate created timestamp in the zone file and a TTL 
set for each dataset you could now actually start decrementing the various 
SOA times as some of them are supposed to. Starting from the initial created 
timestamp not from file reload or anything arbitrary to the local system.

>
> So, I think it's best to introduce another $-special directive, like
> $EXPIRE, to specify expire time.
>
> The only (probably quite serious) problem here is that local time may
> be inaccurate/wrong.

Not that serious from a maintainers viewpoint, if you prohibit/error on 
zones created in the future. Zones expiring early only cause a 
default-accept in the end-systems, which in this case is ironicly better 
than default-deny.

<snip explanation of source for time checks>
>
> So, after some more thoughts, current version of this feature looks like
> this:
>
>  o there's a new directive introduced,
>      $TIMESTAMP when-created when-expires [...probably some more fields]
>    where both timestamps are either in unix time_t format (secounds since
>    Epoch) or in form yyyy-mm-dd[:hh24[:mi[:ss]]] (hours, minutes and
>    secounds are optional), in GMT/UTC.

UTC please.
GMT has english daylight savings problems in some systems.
UNIX time_t gets unweild for manual maintainers (yes we are still out there, 
at least for some zones).

>                                                              When-created 
> must be the current
>    timestamp, and when-expires must be a time when the data will expire
>    and must be reloaded.  When-created is here in order to detect the
>    situation when local time is set to the past, before when-created.

And has other useful benefit side-effects as well. (SOA cycling, etc.)

>  o data expiration is checked during a time when rbldnsd checks for
>    data updates (normally every minute, controlled by -c option).
>    So if you disabled automatic checking for new data (-c0), it will
>    not verify whenever the data has expired during runtime, only during
>    reloads when data has actually changed, which is sorta pointless.
>

Does anyone set -c0 for CPU savings? if not, the timer could still check 
expiry, just not file times.

>  o (optional, I like it but it's somewhat clumsy in implementation,
>    when there are several timestamps per dataset) when expire time
>    is about to come (eg, 90% of time between 'created' and 'expire'
>    has passed), rbldnsd produces a warning on reload.
>

If its seriously clumsy, leave it for later. It might be useful, but we 
don't know for sure yet.
If the zone compiler ever gets done, this should probably be a feature of 
that.

>  o If there are several expire dates per dataset/zone, the smallest one
>    will be used.

Likewise, multiple created lines the _latest_ will need to be used.
So that mixed zones 1hr + 48hr data will be based on the latest 1hr data, 
not the older 48hr by chance ordering.

more below on whole process implications....

>
>  o there's an option introduced, to tell rbldnsd to ignore all those
>    'expire' directives, in order to be able to force it to run "in case
>    of emergency" (current time is wrong, etc).  I dislike introducing
>    this option, but unfortunately there WILL be alot of demand for it
>    once major dnsbls will start using this feature and people upgrade
>    their rbldnsd installs to the version supporting this stuff.
>
>  o Or maybe, an ability to turn this feature on/off per-dataset, by
>    specifying `$TIMESTAMP 0 0'.  Also ugly but demanding...

- not in the same format as a timestamp which is expected.

How so on the 'WILL be a lot of demand for it'?
When the data is old it is invalid and should not be served. End of story.
Even re-purposing the data doesn't change that.
A responsible mirror will adhere to that, an irresponsible one is what this 
feature is designed to avoid.

The only use I could see for keeping old files is someone compiling list 
stats and that doesn't involve rbldnsd.


If the $TIMESTAMP <created> <expires> indicates that data is invalid. Are 
you planning on stopping the file load? or just dumping all the entries 
until another $TIMESTAMP indicates a section of valid data?

I would like to see that behaviour. With rather than the _absolute_ smallest 
expiry time being used, the smallest for the valid data served.

Back to my mixed zone example:
    The 1hr data has failed to regenerate but the 48hr is still valid. Will 
it dump the old 1hr data but still serve the 48hr IPs saying 24 hours to 
expiry?

Perhapse a mixture of the two...
Is it reasonable to expect a reload when data expires and check to see if 
any lines are still valid?
You're thinking if the file has changed the file-mod-time check will catch 
it. Yes it will. But, when the file has _not_changed and only a portion of 
the data is old, the rest still needs to be served.

Other that that bit of extra planning. A very good feature. I am kicking 
myself for not thinking of it and being one of those requests. Especially 
since I spent some days figuring out a way to tell how old my mirror zones 
were and disclaimers to cover it.

Amos Jeffries
Treehouse Networks Ltd.