[rbldnsd] List servers

[Michelle Sullivan]

This is way offtopic for this list so I appologise to all, and request
any followups are to me personally or to another list such as the one I
mentioned earlier.


Scott Haneda wrote:
> On Mar 14, 2010, at 5:55 AM, Michelle Sullivan <matthew at sorbs.net> wrote:
>
>> If you wish to make a public service and if it becomes popular you
>> should prepare for this sort of bandwidth as dealing with the traffic
>> as an after thought, whilst not impossible is significantly more
>> difficult.  You should also be aware that some DNSBls (aka "the
>> competition") will request strongly (demand) they are placed first in
>> the "we've blocked you" priority, and whilst with some MTAs this can
>> mean less lookups to you, it can also mean that you see the traffic and
>> the blocked party never sees your blocked message so you will not get
>> advertising revenue.
>
> I don't entirely understand this statement. You mean that the default
> install of other RBL definition lists get demands to put certain RBL's
> higher up in their config lookup order?

I know some DNSBls will ask to be checked first - if they get a hit then
the other DNSbl's are usually not queried.

>
> For example, SpamAssassain may come pre-defined with 5 known BL's, and
> will have pressure from larger BL's to list those in some form of order?

Doubt spamassassin would take any pressure from anyone. ;-)
>
> I also figured advertising as a model of income was really not there.
> It's all end user eyeballs blind in my opinion. You may get some
> secondary curiosity hitting your website; nothing I would think could
> translate to advertising revenue.

The most advertising revenue I have made on SORBS is $1000/mo ... which
didn't cover the running costs... but some think they can make money
that way.  Of course there are lots of others making money by selling
their lists (or reselling other's lists - in some cases without permission.)
>
> If I were to go at this, it would be free for non business use, and
> price tiered to queries per day for commercial use.

Then you need userbase to make it worth the overheads of accounting.
>
> I always wondered how hard it would be to build the accounting side of
> an RBL with that payment model. How to count lookups based on a
> particular MTA. They can eaily just lookup through a secondary source
> like 8.8.8.8, being a source you could never really block requests
> from. Or perhaps you could, I suppose most rr's don't have a lot of
> business querying an RBL.

Don't know (I'm not interested, I believe it should be free and so far
my employer and the new owner of SORBS concurs.  They think it should be
free for the community.)
>
> Thanks for any general light you can shed on this. It's mostly
> curiosity. While my zone is small I do think it is unique and
> powerful, but may only prove to be so for my particular user base.
>
>> Similarly if a large appliance vendor (or scoring
>> program such as Spamassassin etc) picks up your list you will see a lot
>> of DNS queries with no return traffic for any revenue.
>
> Understood. Though still wondering about my previous question.
>
>> For general interest SORBS' main RBL servers are currently running at
>> 6.5T/month of DNS traffic per server, there are 15 "main" servers.
>
> Holy smokes. I had no idea. Is that all queries? Or I assume that's a
> good deal of rsync as well?

There is other data, but a significant amount is RBL queries.
>
> I always figured rsync would be too latent in data to be truly useful.
> IP's/email addresses/URI's/domains come and go so fast. This entire
> model is built around fast delisting for errors, and even faster
> listing for spammers to be effective.

rsync is useful.  SORBS has implemented rsync deltas.  Rsync now happens
in under 30 seconds so runs every minute.  You can't get faster than
that without a lot of specialist setups.

>
> That means an rsync based system to do local lookups would be pulling
> your entire zone perhaps once every hour or more? Granted, it's only
> the delta that's pulled. Apx what is the delta in MB's in average?
> Assuming you can share that data.
Zone once per day. Deltas every minute.
>
>> That equates to around US$2000 per month in traffic alone for a provider
>> such as Softlayer.
>
> Not a terrible cost. That's bandwidth only though? I assume there are
> colocation costs, amperage considerations, and the hardware?
There is hardware and co-lo in that.
>
> It's not very specialized hardware. Would a rack of many small high
> density yet powrful machines such as Mac Mini's be an option?

Then you are limited on location... You're also not taking into account
infrastructure for the root zone, the website etc..  The front end RBL
servers for SORBS can be 1 RU PIII 1G machines... But the database
servers and the mail servers are massive (20+ blades to handle all of
that) along with several large machines for remote databases (global)
etc..  eg BlackNight our friends in Ireland have sponsored 2 machines (2
cores, 2 cpus each iirc) and they are maxed out handling spamtrap
traffic, DNS and a database backup.
>
> I can rack up 40 or so of those in maybe 10U of cabinet space front to
> back. They put out little heat, use near no power, hold plenty of
> memory... I imagine SSD's could be good in this case, if a machine
> dies, nit a big deal with the redundant nature of this entire system.

I'd estimate SORBS has around 90RU of cabinet space, and whilst some is
wasted with older machines, you try getting a Co-Lo provider to give you
a shelf for mac minis...! ;-)

>
>> Add to that traffic for rsync access, any website (including delisting
>> process and information) and you should realise that starting a new
>> DNSBl is nothing to be taken lightly.
>
> Agreed there. I've had this pipe dream a long time. To do something
> that could give back to a group community I have leeched off of for a
> long time, generally at the cost of no more than what I can donate
> every year.

Consider you could talk to one of the larger providers (spamhaus, SORBS
etc..) about your idea and we might take it on for you.
>
> Every time I think about moving forward, I consider many of the things
> you brought up, and realize it's not a one man operation. I could get
> people to help, but feel you need more. You need people who share a
> passion fir this completely strange, underground, thankless to the
> general public, largely invisible service.

Or get constantly berated and poked fun at ;-)

>
> The thanks you get; "Hey jerk, stop blocking my emails. I'll sue
> you!!!". "see our FAQ, we do not block emails, talk to your ISP".
>
> Speaking of which, does SORBS retain lawyers? Would you say it's
> almost mandatory to do so? Or can some one time legal forms be created
> that can be reourposed?

I had a team of 40 lawyers available to me at all times until 2005, then
I employed my own, now it's not my problem ;-) (that said my last email
to someone with "I'll sue" - only 2 weeks ago - was submitted to the
company lawyers who said, "Yeah, that's right, just change this line
here where your grammar is off"... so I think I do pretty well ;-) )
>
> If this all moved way too OT, let me know (list admins and community)
> and I'll instantly stop and pull this to where Matt suggested.

Matt?
>
> Thanks to rbldnsd for your software to make this even possible, great
> work, amazing track record in stability abd performance. Thank you
> Matt & SORBS ( abd all DNS lists ) for all you do to our inboxes.
>
Regards,

Michelle