The following are blocks of info from a programmer of postfix, 1 of the top 3 used email software programs on the internet. There's some useful links on the page for noobies too, about blacklists and the most common dns errors that will get your email blocked, even if you only sent 1 message. This is how he tells new mail admins to set up their servers, so you can only imagine how much tougher the big 4 are. http://bind8nt.meiway.com/itsaDNSmess.cfm ----------------------------------------------------------------------------------------------- With the ip of the SMTP client: Query DNS for PTR record of ip (reverse lookup) Does the PTR record exist? No: reject mail. Does the PTR record refer to the HELO / EHLO hostname? No: reject mail. Does the ip address's "PTR hostname" have an A record?? No: reject mail. Does the ip address's "PTR hostname" have an MX record?? No: reject mail. Is the ip address in Mail-Abuse.org's MAPS databases of black-holes, open-relays, or dial-ups? Yes: reject mail. Is the reverse delegation the existent and/or correct? No: reject mail. With the HELO / EHLO hostname: Does the hostname resolve to an A record? No: reject mail. Does the hostname resolve to an MX record? No: reject mail. Is the hostname a FQHN: Fully Qualified Host Name ? No: reject mail. If there is an A record, is the ip address in Mail-Abuse.org's MAPS databases of black-holes, open-relays, or dial-ups? Yes: reject mail. With the MAIL FROM: @senderdomain: Does the senderdomain resolve to an A record? No: reject mail. Does the senderdomain resolve to an MX record? No: reject mail. Is the senderdomain a FQHN: Fully Qualified Host Name ? No: reject mail. If there is an A record, is the ip address in Mail-Abuse.org's MAPS databases of black-holes, open-relays, or dial-ups? Yes: reject mail. So, being a responsible and freshly knowledgeable DNS administrator, you're asking "How can I setup my DNS so that my own and my clients' mail is reliably delivered to most restrictive mail servers?" Keep yo bad @ss out of the blocking databases such as Mail-Abuse.org and ORBS. Assuming you aren't a spammer, the most common screw-up is running your mail server as an open-relay. What's that? If you don't know what these anti-spam databases are about, then spend some time studying the pages at Mail-Abuse.org. RFC 2505: Anti-Spam Recommendations for SMTP MTAs RFC 1912 Common DNS Operational and Configuration Errors Make sure your DNS has been delegated with reverse authority for your ip blocks by a higher, wider authority DNS. If your DNS is delegated with reverse zone authority, make sure your reverse zone is setup perfectly, at least for the ip addresses of your mail servers. But why stop there? Just populate your entire reverse zone with real hostnames and generic placeholder hostnames. Check your reverse zone: Is it, or is it not, working? Use the "ip block" and "Scan rDNS" tools at www.SamSpade.org. With the rDNS tool, if you get no results or the results are not what you put in your db. reverse zone file, then your reverse zone is broken. For every domain in your DNS, make sure the zone files contain at least these records: $ORIGIN mydomain.com. @ A ip.ad.re.ss @ MX 10 mailhost.domain.com. For every mail host (real and virtual) in your DNS, make sure the zone files contain at least these records: $ORIGIN somedomain.com. mailhost A ip.ad.re.ss @ MX 10 mailhost.somedomain.com. mailhost MX 10 mailhost.somedomain.com. For every mail host (real and virtual) in your DNS, make sure you know how your mailserver(s) is announcing itself in the HELO / EHLO hostname field. Is your SMTP client sending mail by first announcing itself as: EHLO mail.mydomain.com or EHLO mydomain.com or EHLO smtp.mydomain.com or EHLO (nothing) or EHLO someotherdomain.com ? ? ? ? ? And for whatever the "EHLO/HELO hostname" is: It's a FQHN, Fully Qualified Host Name: host.domain.com. The FQHN has a A record. The FQHN has an MX record. If you get on top of the issues on this page, your DNS won't be a mess anymore. Consequently, your mail and mail servers will have the best chance of not being rejected or blacklisted.
Why is that? If you do so, you will reject a lot of good emails. For example, Launchpad: Code: Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id fs8si3057584wbb.64.2011.10.29.06.25.43 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 29 Oct 2011 06:25:43 -0700 (PDT) indium.canonical.com has only A record and has no MX record - and that's perfectly OK - the sending gateway is not required to be able to receive any emails (it can be an intermediary server after all). Or Google Code: Received: from mail-yi0-f51.google.com (mail-yi0-f51.google.com [209.85.218.51]) by msg.odesk.com (Postfix) with ESMTP id 633F4B905C2 for <[email protected]>; Thu, 1 Sep 2011 03:58:11 +0000 (UTC) mail-yi0-f51.google.com does not have an MX record. Same here: it does not have to have an MX record - sending and receiving domains can be completely different: say, you receive emails to mail.server.com and send it from smtp.server.com MX records are for receiving emails, you are free to send email from any IP address. WHY? MAIL FROM is used to specify the return path - mailbox where all bounces will be delivered to - e.g., [email protected]. myserver.com does NOT have to have an A record (it is NOT an error) but it SHOULD have an MX record - after all, it should be configured to receive incoming emails. Again, senderdomain does not have to resolve to an MX record, it just SHOULD have an MX record. It does not have to
Well I see your points, but a lot of good mail is blocked on different email servers who don't have advanced filtering. So while the points you make are not wrong, I wouldn't be too concerned with what they "should" or "could" allow, don't give any filters a reason is the author's point. Also comparing to google or any whitelist brand is not the same as what the rest of us can do. Point is to check your dns, and if you have your own way that works, then use your way.
Just to clarify something here - I agree with what you said for the most part about not needing MX values for your PTR values, but for anyone that is blocking or weighting your domains based on the presence of MX records, they would most likely be checking the root domain and not a subdomain. If email is coming from a subdomain, and the respective root domain has no MX records...it strongly suggests that the domain has little intention of playing nice...and if you think about it, it is logical that a spammer wouldn't bother setting up MX values because they don't care about processing bounces for FBLs. In the grand scheme of things I don't think a check on the presence of MX values is going to be a major factor in a mailer's ability to send mail. The main thing your DNS server needs to be is reliable and responsive to queries. PTR values are mainly there to reduce the probability that someone is relaying spam through your server...I find that DKIM and SPF records are sufficient, along with PTR values that do not contain "mx" or "mail" in their names.
Good additions dold, if I can ask an opinion about spf, I've been adding them for a range, all ips on that domain in one record with the ~all in it, but I am starting to think one spf record for each ip might be better. I haven't been using bind, I use this mysql program called mydns, so each record is a row in the db, the one per seems like its less obvious if they only query the ip and not every record of the domain. Dkim for sure, domain keys I am starting to think is useless and I notice you didn't mention that. I host my mail servers in a place where they let me manage my own rdns, in a way bad cause its on my mail server which during heavy mailing might ping slowly, but nice cause I was able to auto add rdns with my scripts that add the other records. I also haven't been adding rdns for the www. and domain hosting records, I put like crappy auto built websites on all my mailing domains, and my subdomains host images and masked links. All done auto by my injector. If you have any opinions, would love to hear em, and anyone else who has anything to add, I'd like to hear different approaches.
To put this simply, the main difference between DKIM and DomainKeys is that DomainKeys is attempting to validate the sender domain, whereas the hash in DKIM is based on the contents of the message headers and body itself. You should be using DKIM if anything. As for your question about SPF...I don't think making individual listings would help much either - the purpose of senderID is to validate the IP against the domain it claims to be sending for. What would be ideal, although as far as I know it doesn't have any major impact on mailing as of now, is to be specific enough with your SPF definitions so that you can use the -all rather than ~all. The -all denotes that your SPF record is complete and there are no other IPs that may be sending, claiming that domain as their own. A laggy response or timeout on your DNS server could easily prevent a good chunk of your email from ever being delivered. Not only that, you'd be losing click-thrus as well. Bind is a good choice for high-volume DNS query handling if you tune the settings right. This is what I use, and if it is tuned properly for the volume it will be subjected to, you will not have any perceptible lag during sending. You don't need an PTR record for domains that aren't sending email, but it doesn't hurt to have a PTR record for the root domain even if the IP for the root domain doesn't send email. I do think it is a good idea to have some kind of web presence for the root domain of any domain sending email. You can, at the very least, put a link to your opt-out page and a valid privacy policy. This would help you in the event of a manual review - which happens more often with TLDs and cable domains than with GI.
...or it could be a relay/smarthost. Say, if your MAIL FROM is [email protected] and your email comes from anotherdomain.com it is important that somewhere.com has an MX record; anotherdomain.com does not have to and that's perfectly RFC 5321 compliant. Code: Received: from smtpout.ngs.ru (smtpout10.ngs.ru. [195.93.186.216]) by mx.google.com with ESMTP id gf7si16313429wib.43.2011.12.27.06.53.16; Tue, 27 Dec 2011 06:53:17 -0800 (PST) Received: from [192.168.0.105] (unknown [80.89.192.70]) (Authenticated sender: [email protected]) by mail.ngs.ru (smtp) with ESMTP id 90908111000B for <[email protected]>; Tue, 27 Dec 2011 21:53:11 +0700 (NOVT) Date: Tue, 27 Dec 2011 21:52:49 +0700 From: Someone <[email protected]> In this case smtpout.ngs.ru is a smarthost that relays emails from its users (clients of ngs.ru have their Port 25 blocked*—*this is a common practise to prevent abuse; even antis recommend to do this - e.g., Spamhaus or UCEProtect). smtpout.ngs.ru relays emails to the destination and is NOT configured to receive emails and thus does NOT have an MX record. And if someone rejects this configuration, this is just plain stupid and the administrator of such system should smoke RFC until gets enlightenment (it's logical - just think of Google - its users who use SMTP Submission from their own computer will be rejected with this configuration). It depends. It is the domain in Return-Path header/MAIL FROM email that is responsible for bounce handling after all. And remember, extra DNS check = extra DNS query which means additional load, worse response time. It will be much more faster to query zen.spamhaus.org to reject basing on PBL
Erm, both validate the integrity of the message (that is, headers + body are signed in both scenarios) and both validate the sender. In brief, DKIM does the same as DK plus much more
Thanks guys great input, just testing to see how well I was doing it and I can see some adjustments from your suggestions. Dkim for sure, and the points about the spf was what I was looking for, i didn't know -all existed lol. Sounds better than ~all and I'll stick to 1 record for all. mydns sounded good at first cause I could remotely add records and it doesn't need service restarts but i got this sftp class now that makes it just as easy to add files and my masking and all tracking uses mysql so taking the load off of it for dns might be a big deal. For my own mailing setting up a dns server, as well as a server for just incoming mail is probably ideal once mass volume is going out, the self mail server hosting its own dns is for clients though, who the smaller ones just need an easy way to set things up, and custom set ups for bigger ones. I'm optimizing things now, should be able to move enough stuff around to make the ping times plenty fast but only heavy testing will tell, and I may switch to bind as suggested, flat files are always better than mysql for load. And to add for anyone else who might like this idea, on how to have websites put up on all mailing domains. This concept is what you need, and you can adjust around your needs. Basically all domains point to the same directory, and all hosts to another directory. So your index.php file includes a header file, a content file and a footer file. I use $_SERVER['SERVER_NAME'] to check for the domain name, and it looks for the 3 mentioned include files first. So domain.com.header.php, domain.com.content.php etc, if those files exist it uses those, if they don't it uses a global include file. What this does is, lets every domain instantly have a site with your global files, and then, if desired, you can add those 3 files for a specific domain to override the default and make a totally custom look / content. Also using that server name variable lets you make the content slightly unique each, by making a header and title that say welcome to domain.com or stuff like that. Pretty easy to set up and gives you sites automatically. I have a website builder i made years ago in my interface, so making custom files for a domain is also not too much work, just some template crap.
Being RFC compliant doesn't always mean ideal. There are RFC-compliant setups for many systems that just don't work right in "real world" applications due to issues the RFC does not concern itself with. Spam prevention and related issues are one of those things that the RFC does not concern itself with beyond some minimal surface scratching. The reason for that lies in the fact that the definition of spam is ambiguous. The nice thing about an MX record is that it even if a particular domain is not set up to receive email directly, it can tell the sender where any email should be directed. So smtpout.ngs.ru could simply have an MX record pointing to its email accepting domain. Can you think of one good reason why adding an MX record for smtpout.ngs.ru would be a problem? Simply saying it isn't necessary as per the RFC is a weak justification at best. This is true, but due to the variable nature of mail from & headers and the ease at which they can be manipulated, being able to mail to the exact domain which sent the email provides a layer of assurance that whoever is responsible for the server is getting any feedback that is intended for him. The RFC also mandates that [email protected] should always accept email for the domain in question, relays or not, if it has an SMTP service. There is already a lot of overhead in place due to the spam-antispam struggle. Well thought out caching mechanisms help with this. Yeah that's right. DomainKeys is yahoo's thing, DKIM was based on DomainKeys and expanded upon, then made available as an open standard. Bind can use Mysql if you compile it with support for that, but I wouldn't recommend it for a high volume setup. You're right as far as flat files being better than mysql in this instance. I've been using the one box for all approach and never had any performance issues even with modest hardware.
Of course. The reason is that if smtpout.ngs.ru had an MX record, you could send to [email protected] (well, you can send even to MX-less domains but that's a different story). Since smtpout does not have an MX, it does not have to have a server listening on port 25 (unless it is an MX of some other server) and as a consequence, does not have to receive any emails (after all, your recipients are @ngs.ru, not @smtpout.ngs.ru). Not always. Actually the same RFC says that MX records are not mandatory - if there's no MX, fall back to A/AAAA - they are considered to be an implicit MX record with a preference of 0.
Even though RFCs are not ideal, if you are implementing a filtering system that conflicts with an RFC, you should be prepared that your system will filter out a lot of legitimate recipients.
Right, and you said it yourself. If there is no MX record and email is sent to the domain it would default to the A record...but that's not a good reason not to have an MX record - it's a technicality. Considering how rabid the antis are I don't think citing RFC is going to sway anything in your favor if the domain is engaged in sending high volumes of email, even if you issue a 554 error for attempts to send mail to your smtp out domain. There is no real downside to having an MX record for the email sending domain. It even makes things easier for the admin because all the email sent to subdomains can be collected in one mailbox or sorted according to the subdomain it was sent to. You could even take a page out of Microsoft's book and accept the mail but route it to /dev/null to let the complainers feel like they've accomplished something...but this is getting beside the point. Any good email filtering system will consider multiple factors before determining whether or not a message is spam or not. If you just go by one you will be blocking a lot of legit mail. I personally would not auto-block an email sending server whose domain is without an MX record, but if I could see some anti-spam crusader recommending that the "spam score" for a MX-less domain be increased. For all we know the presence of an MX record may already be a factor for some in their algorithms.
