Cleaning out Bots

Discussion in 'Mail Chat' started by coolbeans, Apr 30, 2012.

  1. coolbeans

    coolbeans New Member

    Joined:
    Apr 23, 2012
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    In front of computer
    I'm working with data that has been mined for clickers/openers, and it's been run through Impressionwise about a year ago, but I still had Webjuice scrub out over 50% of clicks as apparent bot traffic.

    So, I've started trying to track bot clicks by putting a tracking link labeled as "click here" in an html commented out section so that I can gather the offending email addresses to create a scrub list from.

    Is this this best way to reduce bot traffic? Or does anyone have a better way to do this?
     
  2. DAgent

    DAgent Moderator

    Joined:
    Mar 14, 2011
    Messages:
    452
    Likes Received:
    115
    Trophy Points:
    43
    Home Page:
    Bot traffic is not based off the email addresses. Whoever telling you that they remove bots by removing email addresses is lying to you unfortunately. I have bots clicking on my personal gmail/aol/yahoo email accounts.

    Anyone with a client based anti-spam applyance would be seen as bot activity since most of them go through the links to check the landing page.

    Bot clicking must be handled at the email platform level of things, specifically at your redirector. Your redirector must be smart enough to detect if a "click" is coming from a known bot IP and redirect it properly (and not to block it).

    There are threads here that would give you the entire list of known bot IPs for you to check. What you do with that traffic is strictly up to you.

    I hope it helps...
     
  3. nickphx

    nickphx VIP

    Joined:
    Apr 2, 2011
    Messages:
    1,139
    Likes Received:
    363
    Trophy Points:
    83
    Gender:
    Male
    Location:
    guadalajara, chiuhuahua
    You won't be able to remove bots.. Like DAgent said, there's countless software and hardware packages that scan links in email and send them off to be checked.. Your best bet would be to handle bot detection with your link tracking/redirection software.. You can do some simple checks on the user agent, IP address, geo location, or even javascript to weed out the bots and redirect them.
     
  4. coolbeans

    coolbeans New Member

    Joined:
    Apr 23, 2012
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    In front of computer
    Thanks for the advice guys! <( ' ' <)

    And where might I find said lists of known bot IPs?
     
  5. Potterness

    Potterness New Member

    Joined:
    Apr 23, 2012
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
    Thanks for the heads up on this. I've been doing work for a jewelry company and running into similar issues.
     
  6. reddorado

    reddorado VIP

    Joined:
    Jan 24, 2012
    Messages:
    86
    Likes Received:
    33
    Trophy Points:
    0
  7. coolbeans

    coolbeans New Member

    Joined:
    Apr 23, 2012
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    In front of computer
    These appear to be lists to scrub out search engine bots/spiders?

    Or am I not seeing the spam bot list(s)?
     
  8. nickphx

    nickphx VIP

    Joined:
    Apr 2, 2011
    Messages:
    1,139
    Likes Received:
    363
    Trophy Points:
    83
    Gender:
    Male
    Location:
    guadalajara, chiuhuahua
    I bought the maxmind geoip city and org database. I took all of my opens/clicks from the last few months and did lookups on the IPs, storing the results in a database. I then grouped the IPs by org and amount of actions. I was able to find quite a few commercial org ranges that consistently hit my links .. I used that data to build up a nice list of IPs to redirect to some static page.
     
  9. reddorado

    reddorado VIP

    Joined:
    Jan 24, 2012
    Messages:
    86
    Likes Received:
    33
    Trophy Points:
    0
    Yep, mostly search engine bots, but there are some non-search bots under the "Non Spiders" link (http://www.iplists.com/non_engines.txt). We were getting a ton from Cyveillance. But this is by no means an exhaustive list, just a starting point.
     
  10. Broad

    Broad New Member

    Joined:
    May 4, 2012
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    israel
    Home Page:
    we found that detecting if a cookie drop is working within your redirector, filters out efficiently the bots
     
  11. khnet

    khnet Member

    Joined:
    Aug 28, 2011
    Messages:
    44
    Likes Received:
    4
    Trophy Points:
    8
    sounds like the best way to filter out the bots, anyone else using this method?
     
  12. mrlucky123

    mrlucky123 Member

    Joined:
    May 4, 2012
    Messages:
    182
    Likes Received:
    7
    Trophy Points:
    18
    can you elaborate this method? I'm having this problem as well.
     
  13. mrlucky123

    mrlucky123 Member

    Joined:
    May 4, 2012
    Messages:
    182
    Likes Received:
    7
    Trophy Points:
    18
    but with this method, nowaday, many people would disable the cookie in their system though.
     
  14. DAgent

    DAgent Moderator

    Joined:
    Mar 14, 2011
    Messages:
    452
    Likes Received:
    115
    Trophy Points:
    43
    Home Page:
    Place a pixel on your landing page. Robots wont display images while crawling you landing page (if they do -- 95% of them wont). Every click tracked in your system that never reached the landing page, is a bot.
     
  15. DKPMO

    DKPMO VIP

    Joined:
    Mar 31, 2011
    Messages:
    1,452
    Likes Received:
    68
    Trophy Points:
    48
    Location:
    Elaborate Underground Base
    This sounds the same as doing a cookie drop, mentioned above. Right?
     

Share This Page