Fixing Gmail Deliverability Problems By Nicolas Toper and Margaret Morris This is the first in a series of articles we will be running on demystifying some of the deliverability tech stacks out there. Where I work at CritSend, we believe in open sharing of deliverability information, and that deliverability is more of a science than a mysterious dark art. Even though our system fully automates these steps and responses and these are integrated directly into our stack, the bulk of them are repeatable for anyone building their own system. Prerequisites These are essential to basic Gmail deliverability, all of these should be set up: * Track your Open Rates per recipient MX (gmail.com, hotmail.com, comcast.com, etc). Open Rate fluctuation can be a great indicator of problems and/or successes. * Track your Unsubscribe Rates per recipient MX, because Gmail will have different unsub rates than all of your other recipient domains (explained later in this article). * Process your bounces. Gmail especially does not tolerate sending to repeatedly failed addresses. * Sign all of your outgoing mails with DKIM. * Tag or Categorize your different email templates. Knowing which email or list or list segment is causing the problem is key, especially with Gmails complex spam filters. * Provide a list-unsubscribe header in all of your emails. * Make sure your content passes Gmail antispam filters. If you're already using Litmus, run it through that, or you can set up a test address on gmail.com and send your exact message once through your own server to see that very basic content passed. Diagnosing Gmail Trouble Everything is going alright until one day, you see your Gmail Open Rate has plummeted to .5%. So what's the problem? How can you solve it? How could you have caught it sooner? First of all, there are two warning signs: * If you're using a spambox monitoring system such as the ones embedded in CritSend or Return Path, you'lll be able to detect the issue by seeing increase in Spambox Placement. * A few percent drops over time in the Open Rate. That's the last time to act! Gmail has started putting your email in the spambox if your users do not complain, it will put most of your emails there. Before we discuss solutions, let's peek into Gmail's spam filter is working. Overview of the Gmail Spam Filter Gmail spam filters have 3 significant components: 1. Content. It relies mostly on DKIM keys AND your email template. Gmail's content filter is a template-based content filter fed in real time by user reaction and user engagement. It has the capacity to redirect a specific template straight to the Spam Box, and even evade the most pristine good-reputation DKIM keys. Gmail scores a DKIM key AND a given template (in contrast, Hotmail scores mainly your template and IP). 2. Engagement. A real-time behavioral user engagement check. If a lot of your users are clicking on their Report Spam buttons, your email will land in spam. Make sure you have tracking and alerts set up for negative or declining user engagement in real time so you can act quickly. Depending on the seriousness of the issue, this filter will score your DKIM key and your email template and send subsequent deliveries into "spamland". 3. SMTP. An IP blocking system. This is really only used by Gmail in cases of severe repeated harsh spam. More antiquated systems like Hotmail are more likely to trigger IP blocks, but they are even phasing those out more. If a person sends Viagra mail, it will land in the Spam Box straight (content filter). If as an Email Service Provider you accept the "email kit" of a known spammer, it will land in the Spam Box straight (content filter). If a phisher sends a "smart'" phishing email it will start into the inbox, then later land in the Spam Box and all their mails (meaning mails sharing the same DKIM key) after that will end up in the Spam Box (content and engagement filters). But now more than ever, Spam is whatever annoys your recipients. So say you're a social network developer who allows its users to send invites to their entire contact list and those invites are badly worded or your users are repeatedly sending to the same people, they will eventually get annoyed and click on their Spam Report button. The first few times this happens, it's no biggie, but if it happens a large percentage on one template, Gmail will land it straight into spam. Gmail will protect its end users and start landing all your subsequent mails into spam. There are tons of other cases and examples, but just know that taking the prerequisite measures listed above will allow you to detect all of those before they escalate into issues. At CritSend our Deliverability Engine automates all of these detection services and handles all prerequesites to prevent any deliverability fluctuation from escalating to a full fledged problem. Solutions So what if you're already IN the Spam Box? 1. The first step is to identify whether your email template or/and or your DKIM are blacklisted. Figuring that out is pretty easy as you just need to test it on one or two test accounts. Then change accordindly your DKIM key and/or your template. Unless blatant spam, usually you can just change your DKIM key. 2. Finally you need to fix the true source of the problem. If you fail to remedy the source of the problem, your brand new shiny DKIM key will soon go into the Spam Box too. If you're sending out multiple email templates, then you need to identify the problematic email asap. Usually it's the template with the highest unsubscription rate. Why? Gmail offers no Feedback Loop. Therefore, when a Gmail recipient clicks her Report Spam button, Gmail doesn't report that to you or your ESP. Doesn't mean it doesn]t matter, they just don't TELL you. So how do you find out who clicked on Report Spam? Simple. When a Gmail recipient clicks on the Report Spam button on any mail that has a list-unsubscribe header, Gmail ALSO shoots that Gmail user a popup oSo even though Google doesn't offer anyone a true feedback loop, you can simulate a very useful FBL-type indicator. Then you can infer this into your Spam Report Rate. So even though Google doesn't offer anyone a true feedback loop, you can simulate a very useful FBL-type indicator. Then you can infer this into your Spam Report Rate. We hope we have shed some light on the ever-evolving Gmail Spam Placement and Block magic. Source: http://blog.deliverability.com/2011/11/fixing-gmail-deliverability-problems.html
This is very good stuff, gmail can be really difficult and with more and more GI switching to it, it will become even more important to learn it. I've started looking for their mx records in GI domains and attaching that domain to gmail settings, so that domain doesn't get treated like real GI.
Are you also hesitant about inserting the header "Date" as it may imply that you want to go out with complete strangers? Here I will throw you a bone: http://www.ietf.org/rfc/rfc2369.txt This header is all pro and no con.
This is an important header for gmail deliverability as it essentially performs the same function of a real fbl, and that is removing these people from your lists.
As far as I know, gmail is the only one currently making use of this header. From what I've seen on other platforms, this header does not negatively effect anything.
My take on this ... if the isps know you are sending bulk, which they most likely do, then you would add the unsub header, showing them you care about removals. If you thought however, you were slipping email past without being tagged as bulk, then the unsub header screams to them that you are sending bulk. So yes, in most situations adding them is all pro and no con as Solarus said, but its not 100% to everyone, depends on how you feel they are seeing you. Now for you SG, if you're headers are going to be the same for all emails in a broadcast, 2 things: 1. They definitely know you are sending bulk. If headers don't change, I assume your content is also the same, so there is no way they will miss all your other emails and not see you as a bulk sender. This is normal, so I wouldn't worry about it, but if you accept that, then adding the unsub header should be easier to accept as well. 2. If you had to add the unsub header, just for gmail, and risked telling them you are sending bulk, it would be worth it. Having a fbl for google is huge.
I assume you aren't inboxing, Are you 100% junking? Getting in 100% of your gmail? If NO, then you're broken and you should try IMO.
It's not really about fixing anything that's broken, more like a feature add. Your system already has the ability to handle fbl's, why not have something in place that can virtually do the same with gmail, and get rid of complainers?
Ya I agree with fun 100%, its a new feature not a fix. Fbls are huge and gmail is rough not having a standard one, anything you can do to remove people who will report you as spam is a huge thing. As a programmer too I will suggest this, you don't need to have this installed by tomorrow or anything, so I would start a new update of the software by adding in some specific isp settings, including a custom header option for an isp you define. Not to make the feature specfic because if you make it general like that it leads to lots of options later instead of just this one, and then you, or your clients will have the ability to add a custom header definition for any domain they want, and they, or you, can add a list unsub to gmail to get use of this feature, as well as any other headers they may want to add for other domains. Its all about flexibility, no reason not to have this as an option at the least.
Never say guarantee!! lol. And this isn't necessarily true, like I said earlier you already are seen as bulk by volume, so this header can actually make you look more legit, there's all kinds fo bulk senders and most get group up as spam, but it can't really hurt if you are already junking. I would say we can all argue this point all day, won't tell you anything, you will have to split test it.
Yes, they are giving the option of unsub'ing the user because they will click the link for them. Its pretty much a full fbl if you set it up right.
Understood, and I can see where you are coming from on this. I call it "protecting clients from themselves" lol, they mess with something they don't understand, mess up and blame you for it. Always a balance of adding features and keeping things organized, your call on that. But cool, sounds like you have your plan, glad I could help.
This is one of the best threads I've seen in a long while. Thanks guys, I quite enjoyed the co-operative spirit here. Oh and thanks SG "The Mad Genius" for improving on our mailing system even more
It is not Google invention. This tag is independent industry standard and I believe every major webmail and ESP implements it. Even frigging Interspire adds it by default.
I believe they did. Did you find out otherwise? Hotmail apparently does: http://www.aweber.com/blog/email-deliverability/list-unsubscribe-header.htm
Well I hate google for many reasons lol, but this is nice for the moment. Don't get too comfortable with it though, because once google gets around to making the same type of annoying fbl system the others use, they will. I don't know why they havent yet honestly, its probably their idea to not promote bulk mailers, but as the others have realized, we aren't going away and the formal annoying sign up (from their perspective, obviously not ours) is a good way to slow us down. Waiting for those confirmations makes it so we can't open up and blast on hundreds of domains without some work and waiting. (unless we ignore fbls, which is also not wise for a long term plan).
I do not think Google sees the benefit to themselves from offering FBL. They want to keep raw spam/not-spam feedback to themselves and use it for filtering. Apparently they believe that not allowing mailers to directly their scores could give them more precise data on who to inbox vs. bulk vs. block. So I would not hold our breath for Google to offer FBLs. If anything I'd be concerned that others would drop FBLs, the only thing that's stopping them is they are not as good at using recipient feedback to figure inbox placement.
