Hi everyone --- I'm new to MF, but I've been doing high volume email marketing for 8+ years - on my own, at an ESP, and at a very large list manager. I was wondering if this forum is game for some tech discussion or if this is not the right crowd. I'm in the middle of redoing our infrastructure from using commercial MTAs to using Postfix. I've learned a lot of cool tricks about how to get a single instance of Postfix to manage multiple outbound IPs, how to architect it to throttle and rate limit per ISP, etc. Happy to share some of that knowledge and would love to swap ideas for anyone managing their own deliverability infrastructure. Anyone interested? --Red
Thanks Dude. It's nice to finally find a mailing forum that's not chock full of wanna-be black hat idiots trying to get rich off of harvested data. Refreshing...
most def interested in tech talk here and sharing your knowledge is a surefire way to build street cred here. Welcome aboard. Jedi
We are running a mix of PMTA and Message Systems today, but didn't feel like we were getting value for the money in terms of license costs. Don't get me wrong - they are both great systems (each with pros and cons). But we're looking for a solution that's (1) cheaper to scale across many nodes and (2) compatible with any host. We looked at a bunch of open source options including sendmail, exim and postfix. Our issue with Postfix in the past was that you had to run one instance for every outbound IP you had - and it crushed boxes that we want to put a lot of IPs on. The game changer for us was Postfix 2.7 (the standard install is still 2.3.3 on many linux distros). 2.7 has a feature called sender_dependent_transport_maps. It makes it very simple to inject into Postfix, and use the envelope sender to tell postfix which outbound IP to use. So now we can have hundreds or thousands of IPs managed by a single instance. Obviously you're not going to get the throughput per box that you would get with a commercial bulk MTA. But we've been surprised at how well postfix has done. Particularly if you separate inbound bounce handling from your outbound MTAs (which saves you on inodes). If anyone has tried and failed to configure their own postfix infrastructure, I am happy to help. It's like trying to solve a rubiks cube with your eyes closed the first time and then once you figure it out, it's much easier to do! Curious how many people here are running their own infrastructure vs using ESPs?
A huge problem with postfix is that it will hit you back hard with your own bounces and you cannot turn that off. I found this impractical. Are you still looking to move to it or already have it in production? Do tell.
I've never tried to run postfix in a bulk mailing environment let me know how it works out for you. How do you plan to address back off and rate limiting at the ip to domain level? For example ip 1.1.1.1 is set at 100 emails an hour for aol, but 200 an hour for yahoo. Where have you tried to install PowerMTA at that you can't? Why do you feel you aren't getting value for you money?
Impractical is very true, although I do believe you can turn off the bounce queue and you certainly should if I am correct in this. They may have added this since you last tried, if I remember correctly there is a setting for how many retries you want to send, and you set it to 0.
The volume is all on how well you can thread it, and I know a few people who couldn't get more than 50k messages out an hour because they couldn't thread it. But the OP addressed this and said he has it threading, in which case it can send as much mail as a bulk mta. Back off is a problem, there may be setting somewhere that I don't know about, but I never found anything for immediate halt like on a TSO, maybe there's a mod or you would have to write that in. And would have to be directly in postfix, finding where its looping through the queue directory look for a halt file or something like that. The settings should be in the injector to set up for each socket, I found this to be a bitch with postfix as you can make the settings globally but I stopped playing with postfix before getting on the fly settings but the idea I spoke to a few others who had tried postfix was fake headers with the settings info, stripped out before sending out and used as settings. I have no idea how this is done however but it was the only method I heard of that could pass info to the mta. This was the point when I said forget this, and wrote a smtp wrapper that let me pass exactly what settings I wanted per socket, including which mx ip I sent to and which ip to bind to. It threads for each connection to send concurrently as many connections I pass to it, which ip to send on, host, domain, throttle and a directory to loop through of all emails for each session. Changes and additions to your own code is cake as its meant to act the way you build it to instead of having to work around what the program you are using wants to do. Then your injector has freedom to do whatever you can think of.
I've never used MessageSystems software.. I have used PowerMTA extensively without any trouble.. I would consider switching from powermta to postfix a serious downgrade.. If you have a c++ monkey on staff then I could see how it would be worthwhile.
I never said anything about how to do volume . I was more curious how he planned on doing ratelimiting and backoffs. I know how I would do them and it's similar to your suggestion. I'd modify the code if I couldn't find a patch for backoffs and I'd do the following for ratelimiting (might be wrong I've never done this, but it seems like it would work and if it does it's an ugly hack). Injection Server (referred to as IS) -> Outbound Server 1 (referred to as OS1..Only Mails Yahoo.com) Inject Server( referred to as IS) -> Outbound Server 2 (referred to as OS2..Only Mail Aol.com) With having each out bound server* targeting a unique TLD it should be a lot easier to control the rate limiting for that TLD by using the incoming rate limiting settings on OS1 and OS2. Injection would be done via IS with rules to route certain TLDS via the respective OS or via SMTP feeding directly to the respective OS with retry rules for when it hit the ratelimits imposed by the respective OS. I agree with nickphx though. Postfix seems to be a down grade from pmta. *By server it doesn't have to a physical server. It can be another postfix instance. However, with the addition of the feature mentioned by OP you might be able to set the incoming rate limit on a per ip basis.
You can turn off or tweak only a few minor features of the bounce queue. I played around with the queues awhile back and found that anytime postfix gets a 4xx or 5xx reply it is hardwired to generate a bounce notification to the postmaster. The software author specifically said on forum this can never be turned off. In practice this means if you are having any kind of delivery hiccup your whole systems would get quickly gummed up by Postfix trying to deliver your entire mail back to you. Postfix was designed mostly for people who want to mainly receive/filter mail for a very small number of mailboxes. When you try to send lots of mail it just breaks down.
Oh I see what you mean, I didn't realize you meant the sending of a bounce email, ya that would be pretty bad if you couldn't stop it. And you're right about the design of postfix, its a great program for normal mail but just is lacking all the important stuff needed to control bulk, its certainly possible but I think if you are good enough to modify it you could probably make a bad ass smtp wrapper too
I think if you do have the capability to go to source code and hack postfix it might be a better idea to just scrap it and build a better sending system from scratch.
