Automated suppression list downloader

Discussion in 'Mail Chat' started by phdesign, Jul 2, 2014.

  1. phdesign

    phdesign Active Member

    Joined:
    Dec 12, 2011
    Messages:
    123
    Likes Received:
    32
    Trophy Points:
    28
    Hey all,

    Just wanted to know if, outside of custom programming, there is a program that can download suppression files weekly from Optizmo, Unsubcentral, DNElist, the rest of these suppression companies, ftp sites (logged in or not), etc.? All the better if it can rename the downloaded files.

    I spoke with Optizmo and Unsubcentral a while ago and while they will download files from FTPs and networks using their system, neither wanted to grab each others files. DNElist says they work with those two, but not sure they can grab files from all the others out there.

    I've tried some automated FTP apps and other tools but nothing seems to be able to get all the files from all the different networks and advertisers. None of the regular apps will tap into APIs. But even if an app could download 80%+ it would be a bonus.

    I have worked with an MTA that was programmed for each API, FTP, etc. Since I'm not using that one now, I'm looking for something standalone for Windows or even something server side for Linux. Otherwise, I'll be paying another programmer to build this again. I'm sure this has been built and is probably out there. I'm sure some of you already have done this and I'd pay for something not too overpriced ;)

    Thanks
     
  2. DKPMO

    DKPMO VIP

    Joined:
    Mar 31, 2011
    Messages:
    1,452
    Likes Received:
    68
    Trophy Points:
    48
    Location:
    Elaborate Underground Base
    Some of these suppression systems have APIs to download their files.

    Having implemented the major ones I can say doing this properly is a significant undertaking.

    Either way all this is pretty useless unless you can actually automate the import of these files into your mailing system and then deal with deluge of this data and all sorts of issues that come with its deduping, handling emails vs. domains and all sorts of weird error handling for each of different suppression file sources...

    Nobody is going to automate this for you unless there is money to be made. How much would you pay for this?
     
  3. phdesign

    phdesign Active Member

    Joined:
    Dec 12, 2011
    Messages:
    123
    Likes Received:
    32
    Trophy Points:
    28
    Yeah, I wouldn't expect this for free. I was hoping there might be some service that did this already for an interim solution to save 8-10 hours of work per month.

    Some of the networks using platforms like Cake have a pretty simple call for grabbing the files. And getting basic FTP automated is pretty simple if the file name is the same all the time. However, as you said it isn't always just that simple working with some of the suppression systems. I've looked at the APIs for a few of them that aren't so easy (have to build the files on the fly for example and download once they are built) and what is done with the files when they are retrieved (I want them renamed, unzipped, and FTP'd to the MTAs for example) will take some extra steps as well.

    Right now I use three MTAs for various things. One of them handles most of the suppression automatically as long as the offers are from networks with platforms that are integrated (I use two that are not so even there I could use some automation). Once I set up a campaign, it retrieves the file(s) at least weekly and is hands off. If the file isn't updated for whatever reason within 5 days, it won't send the campaign and lets me know. This works so well I'd like to have it on the other systems.

    The second MTA will probably get it done in the next 12-18 months (nowhere near the top of the to-do list).

    The other one is my own and I'll have to get it programmed myself regardless, but I have a few other things taking priority right now.

    I was going to pay Optizmo a few hundred per month until I realized they wouldn't import the files from the competitors. Just having Optizmo might save a few hours per month, but it makes more sense programming something to work with the major suppression providers, networks, etc..

    I don't really want to pay hundreds per month for years to come when I can pay $1000 or $2000 to get it integrated with at least the networks and advertisers I currently work with. However, I'd pay some monthly fee for a few months right now if it was already built. And if you're suggesting you might sell the source of what you have, let me know. ;)

    Sounds like I may be mapping this out for a programmer...
     
  4. DKPMO

    DKPMO VIP

    Joined:
    Mar 31, 2011
    Messages:
    1,452
    Likes Received:
    68
    Trophy Points:
    48
    Location:
    Elaborate Underground Base
    "A pretty simple call for grabbing the files" is the easiest part. A lot of suppression sources are a mess and stuff constantly changes. Re-downloading huge suppressions every week is another issue. That's why people host their files in systems like Optizmo because it is a lot easier to use them to download only changed records. etc. etc. etc.

    If you want to pay just a few hundred bucks at most, I doubt it makes any sense for us to branch off our code into a standalone product... We won't be sharing any source code either. The functionality will be available as part of the larger system though and at some point revealed publicly.
     
  5. phdesign

    phdesign Active Member

    Joined:
    Dec 12, 2011
    Messages:
    123
    Likes Received:
    32
    Trophy Points:
    28
    Ok, folks... reviving a cold post here, but it is mine ;) And some of you might find this helpful.

    I wanted to share something that helped me, a non-programmer, automate some time consuming tasks. As I originally mentioned, I was looking for something that could, even temporarily, automate things like downloading suppression files. What I needed was to download the files from various networks, in some cases rename or rework them a little (noted below XLS to TXT), and upload them to my sending servers and such.

    I happened to Google into a few softwares for Windoze that automate a handful of these kinds of tasks. I was looking for something that could simply schedule FTP downloads so I could automatically grab suppression files from network FTPs. What I found was far better in that it could download not only from FTP, but from web links and more, and it could manipulate the files and upload them to multiple servers. The one I decided to test out is called WinAutomation.

    In about 4 hours I had it trained to download around 70 or 80 files from a handful of networks. It worked with a Cake network that had the files on a passworded FTP, it worked with Cake using Optizmo (I trained it to click the button, wait for the next page and download from the link), non-passworded files on links, Linktrust, and and a few others. It did not work with Hitpath, at least not yet (only had 6 files on that network so didn't spend too much time there). I didn't try Unsubcentral, DNElist or any others like that yet.

    Today I added two offers and it took a few minutes to set it up for the new suppression files (mostly copy/paste the existing tasks and rename the files and such). And now they're in the routine and I will never have to manually touch those files again unless something changes with them or I drop the offers. It takes around an hour to run through all the lists... but only takes a couple of clicks for me and my computer does the rest. It occasionally "breaks" if a filename changes or is missing, but otherwise it runs smooth.

    I set it up to download the files, unzip them, rename the inner files and upload them to my servers. I also trained it to open Excel files, grab the column with the emails, copy them, paste them into notepad, save the notepad file with the new name, then upload those .txt files to the servers.

    It also grabs two daily feed data lists does a few things there and uploads them to the servers. The servers auto import so that wasn't a big deal to me but you could perhaps train a macro to click the buttons on your sender and import them.

    I'm a big fan of automation. Spending $150 and 4 hours of time to save countless hours of mundane repetition has been a big help for me. I'm sure I'll find some other uses for this software as well. No, I'm not being paid to talk about it and won't even provide a link... just Google WinAutomation if you're interested.

    For me this was a short-term solution as I'm still going to have a programmer integrate this directly for me. But maybe this will be useful for others here.
     
  6. Daansen

    Daansen Member

    Joined:
    Aug 14, 2011
    Messages:
    84
    Likes Received:
    16
    Trophy Points:
    8
  7. ca1980

    ca1980 VIP

    Joined:
    Jan 10, 2013
    Messages:
    26
    Likes Received:
    2
    Trophy Points:
    3
    Most of the networks use similar backend software, so I usually just hire a cheap programmer to write a CURL script to login and download the supp lists at regular intervals, and upload to my mailing platform. Costs like $30 and saves a lot of grunt work. Only gets annoying when networks change things up, which doesn't happen too often.
     

Share This Page