[rbldnsd] enhanced dnset

[Sami Farin]

On Sun, Nov 27, 2005 at 11:13:20AM +0100, David Landgren wrote:
> And Sami Farin did write:
> >On Sat, Nov 26, 2005 at 12:07:17AM +0300, Michael Tokarev wrote:
> >....
> >
> >>>if you don't want to give TXT record, use
> >>>dls.net :\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3}\.dls\.net
> >>>(PCRE pattern is always the latest field delimeted by ':').
> >>>
> >>>Does this sound sane?
> >>>Free tips'n'tricks?
> >>
> >>Well, I don't think it's necessary to invent new complicated syntax
> >>for such stuff.  Plain regexps (maybe modified a bit to be more easy
> >>for domains where a dot (.) is commonly used) are just fine, ie,
> >
> >
> >I mentioned PCRE because I have used it before... 
> >(I added PCRE support for qmail).
> >But one pcre_exec takes only around 3000 CPU cycles for
> >the patterns like those mentioned in this email.
> >
> >2GHz CPU could do those 666666 a second.
> 
> I have written a Perl module (Regexp::Assemble [1]) that will allow you 
> to take an arbitrary number of individual regexps and compile them down 
> into one expression.

I have heard of this kind of software before but didn't remember
when I was thinking of PCRE+rbldnsd, good that you refreshed my
memory.

> You could then feed that single regexp to 
> pcre/rbldnsd for use.
>
> I do this already for DNS hostnames for residential addresses, like 
> 1-2-3-4.bogo.example.com. I currently have about 3500 distinct regexps 
> (the above example would become \d+-\d+-\d+-\d+\.bogo\.example\.com) 
> which compile down into a single pattern of about 80 kb.

How many CPU cycles does pcre_exec take for that long a pattern?

In your test data you have for example many *.dial.brasiltelecom.net.br
but I don't need to include those in the pattern, that can be matched
with rbldnsd dnset.  Now my dnset contains  32156 lines and
one "more aggressive one" 105679 lines, of which 65536 are needed
to cover one particular ISP :) , and for many ISPs I need to list
*.0.idiotisp.biz ... *.255.idiotisp.biz.  

I expect the one-expression-PCRE-pattern I need in rbldnsd would be
comprised of at max some hundred domains.

You have included really neat command-line util assemble in [1]
which I believe makes it easy to create the pattern to be used
with rbldnsd.  Now it's left to be decided how to implement PCRE
into rbldnsd and where/how to configure them.
I guess I could just give out the configurable TXT record and
additionally the captured substring.

> You do have to recompile pcre and up the LINK_SIZE define if you want to 
> play around with big patterns, though.

Good tip... I try to remember that.

> Later,
> David
> 
> 1. http://search.cpan.org/dist/Regexp-Assemble/

--