I'm new to MF - are you up for tech talk?

Discussion in 'Noob Central' started by reddorado, Jan 25, 2012.

  1. reddorado

    reddorado VIP

    Joined:
    Jan 24, 2012
    Messages:
    86
    Likes Received:
    33
    Trophy Points:
    0
    Hi everyone ---

    I'm new to MF, but I've been doing high volume email marketing for 8+ years - on my own, at an ESP, and at a very large list manager.

    I was wondering if this forum is game for some tech discussion or if this is not the right crowd. I'm in the middle of redoing our infrastructure from using commercial MTAs to using Postfix. I've learned a lot of cool tricks about how to get a single instance of Postfix to manage multiple outbound IPs, how to architect it to throttle and rate limit per ISP, etc. Happy to share some of that knowledge and would love to swap ideas for anyone managing their own deliverability infrastructure.

    Anyone interested?
    --Red
     
  2. DaMadHatter

    DaMadHatter Active Member

    Joined:
    Mar 1, 2011
    Messages:
    720
    Likes Received:
    51
    Trophy Points:
    28
    Location:
    In the Void
    Welcome aboard buddy.

    :thrasher:
     
  3. reddorado

    reddorado VIP

    Joined:
    Jan 24, 2012
    Messages:
    86
    Likes Received:
    33
    Trophy Points:
    0
    Thanks Dude. It's nice to finally find a mailing forum that's not chock full of wanna-be black hat idiots trying to get rich off of harvested data. :) Refreshing...
     
  4. emailjedi

    emailjedi VIP

    Joined:
    Apr 26, 2011
    Messages:
    138
    Likes Received:
    7
    Trophy Points:
    18
    Location:
    The Outer Rim
    most def interested in tech talk here and sharing your knowledge is a surefire way to build street cred here.


    Welcome aboard.

    Jedi
     
  5. JohnFarrell

    JohnFarrell VIP

    Joined:
    Apr 13, 2011
    Messages:
    828
    Likes Received:
    35
    Trophy Points:
    0
    What are you moving away from commercial mtas to postfix?
     
  6. reddorado

    reddorado VIP

    Joined:
    Jan 24, 2012
    Messages:
    86
    Likes Received:
    33
    Trophy Points:
    0
    We are running a mix of PMTA and Message Systems today, but didn't feel like we were getting value for the money in terms of license costs. Don't get me wrong - they are both great systems (each with pros and cons). But we're looking for a solution that's (1) cheaper to scale across many nodes and (2) compatible with any host.

    We looked at a bunch of open source options including sendmail, exim and postfix. Our issue with Postfix in the past was that you had to run one instance for every outbound IP you had - and it crushed boxes that we want to put a lot of IPs on.
    The game changer for us was Postfix 2.7 (the standard install is still 2.3.3 on many linux distros). 2.7 has a feature called sender_dependent_transport_maps. It makes it very simple to inject into Postfix, and use the envelope sender to tell postfix which outbound IP to use. So now we can have hundreds or thousands of IPs managed by a single instance.

    Obviously you're not going to get the throughput per box that you would get with a commercial bulk MTA. But we've been surprised at how well postfix has done. Particularly if you separate inbound bounce handling from your outbound MTAs (which saves you on inodes).

    If anyone has tried and failed to configure their own postfix infrastructure, I am happy to help. It's like trying to solve a rubiks cube with your eyes closed the first time and then once you figure it out, it's much easier to do!

    Curious how many people here are running their own infrastructure vs using ESPs?
     
  7. DKPMO

    DKPMO VIP

    Joined:
    Mar 31, 2011
    Messages:
    1,452
    Likes Received:
    68
    Trophy Points:
    48
    Location:
    Elaborate Underground Base
    A huge problem with postfix is that it will hit you back hard with your own bounces and you cannot turn that off. I found this impractical.

    Are you still looking to move to it or already have it in production? Do tell.
     
  8. JohnFarrell

    JohnFarrell VIP

    Joined:
    Apr 13, 2011
    Messages:
    828
    Likes Received:
    35
    Trophy Points:
    0
    I've never tried to run postfix in a bulk mailing environment let me know how it works out for you.

    How do you plan to address back off and rate limiting at the ip to domain level? For example ip 1.1.1.1 is set at 100 emails an hour for aol, but 200 an hour for yahoo.

    Where have you tried to install PowerMTA at that you can't? Why do you feel you aren't getting value for you money?
     
  9. afc_pmc

    afc_pmc New Member

    Joined:
    May 30, 2011
    Messages:
    192
    Likes Received:
    17
    Trophy Points:
    0
    Impractical is very true, although I do believe you can turn off the bounce queue and you certainly should if I am correct in this. They may have added this since you last tried, if I remember correctly there is a setting for how many retries you want to send, and you set it to 0.
     
    Last edited by a moderator: Jan 27, 2012
  10. afc_pmc

    afc_pmc New Member

    Joined:
    May 30, 2011
    Messages:
    192
    Likes Received:
    17
    Trophy Points:
    0
    The volume is all on how well you can thread it, and I know a few people who couldn't get more than 50k messages out an hour because they couldn't thread it. But the OP addressed this and said he has it threading, in which case it can send as much mail as a bulk mta.

    Back off is a problem, there may be setting somewhere that I don't know about, but I never found anything for immediate halt like on a TSO, maybe there's a mod or you would have to write that in. And would have to be directly in postfix, finding where its looping through the queue directory look for a halt file or something like that.

    The settings should be in the injector to set up for each socket, I found this to be a bitch with postfix as you can make the settings globally but I stopped playing with postfix before getting on the fly settings but the idea I spoke to a few others who had tried postfix was fake headers with the settings info, stripped out before sending out and used as settings. I have no idea how this is done however but it was the only method I heard of that could pass info to the mta. This was the point when I said forget this, and wrote a smtp wrapper that let me pass exactly what settings I wanted per socket, including which mx ip I sent to and which ip to bind to. It threads for each connection to send concurrently as many connections I pass to it, which ip to send on, host, domain, throttle and a directory to loop through of all emails for each session.

    Changes and additions to your own code is cake as its meant to act the way you build it to instead of having to work around what the program you are using wants to do. Then your injector has freedom to do whatever you can think of.
     
  11. nickphx

    nickphx VIP

    Joined:
    Apr 2, 2011
    Messages:
    1,139
    Likes Received:
    363
    Trophy Points:
    83
    Gender:
    Male
    Location:
    guadalajara, chiuhuahua
    I've never used MessageSystems software.. I have used PowerMTA extensively without any trouble.. I would consider switching from powermta to postfix a serious downgrade.. If you have a c++ monkey on staff then I could see how it would be worthwhile.
     
  12. JohnFarrell

    JohnFarrell VIP

    Joined:
    Apr 13, 2011
    Messages:
    828
    Likes Received:
    35
    Trophy Points:
    0
    I never said anything about how to do volume :p. I was more curious how he planned on doing ratelimiting and backoffs. I know how I would do them and it's similar to your suggestion. I'd modify the code if I couldn't find a patch for backoffs and I'd do the following for ratelimiting (might be wrong I've never done this, but it seems like it would work and if it does it's an ugly hack).

    Injection Server (referred to as IS) -> Outbound Server 1 (referred to as OS1..Only Mails Yahoo.com)
    Inject Server( referred to as IS) -> Outbound Server 2 (referred to as OS2..Only Mail Aol.com)

    With having each out bound server* targeting a unique TLD it should be a lot easier to control the rate limiting for that TLD by using the incoming rate limiting settings on OS1 and OS2. Injection would be done via IS with rules to route certain TLDS via the respective OS or via SMTP feeding directly to the respective OS with retry rules for when it hit the ratelimits imposed by the respective OS.

    I agree with nickphx though. Postfix seems to be a down grade from pmta.

    *By server it doesn't have to a physical server. It can be another postfix instance. However, with the addition of the feature mentioned by OP you might be able to set the incoming rate limit on a per ip basis.
     
    Last edited: Jan 27, 2012
  13. DKPMO

    DKPMO VIP

    Joined:
    Mar 31, 2011
    Messages:
    1,452
    Likes Received:
    68
    Trophy Points:
    48
    Location:
    Elaborate Underground Base
    You can turn off or tweak only a few minor features of the bounce queue.

    I played around with the queues awhile back and found that anytime postfix gets a 4xx or 5xx reply it is hardwired to generate a bounce notification to the postmaster. The software author specifically said on forum this can never be turned off. In practice this means if you are having any kind of delivery hiccup your whole systems would get quickly gummed up by Postfix trying to deliver your entire mail back to you.

    Postfix was designed mostly for people who want to mainly receive/filter mail for a very small number of mailboxes. When you try to send lots of mail it just breaks down.
     
  14. afc_pmc

    afc_pmc New Member

    Joined:
    May 30, 2011
    Messages:
    192
    Likes Received:
    17
    Trophy Points:
    0
    Oh I see what you mean, I didn't realize you meant the sending of a bounce email, ya that would be pretty bad if you couldn't stop it. And you're right about the design of postfix, its a great program for normal mail but just is lacking all the important stuff needed to control bulk, its certainly possible but I think if you are good enough to modify it you could probably make a bad ass smtp wrapper too
     
  15. DKPMO

    DKPMO VIP

    Joined:
    Mar 31, 2011
    Messages:
    1,452
    Likes Received:
    68
    Trophy Points:
    48
    Location:
    Elaborate Underground Base
    I think if you do have the capability to go to source code and hack postfix it might be a better idea to just scrap it and build a better sending system from scratch.
     

Share This Page