Hi, I have some free time I want to use to learn a programming language well. The long-term goal is to create my own feeder for postfix. So the question is what should I learn. I have almost no knowledge at this at all, but still my goal isn't to come to a point where I notice all the coding I have done isn't worth it because the language I used is too slow or limited. I want to build up a lot different features (tracking, autounsubs, etc.) rather than have it look pretty. The two viable choices according to my research up to now are: - Perl - Php with MySQL - Java (not sure about that one The prefered server it would be running on is Linux Debian/ubuntu. I heared to achieve an app that send email is probably fastest done in Perl, but I also heared perl is pretty slow as well as php. Suggestions are greatly appreciated.
I havent done a feed for postfix, but I have done one for PowerMTA and I imagine the concepts are similar. In response to what language you should use, the answer is yes. I used all four for my application. MySQL for the database (because its free and sufficient) This is your workhorse and will be your biggest issue. When you are only doing a few hundred thousand data points its easy, but as you start to manage millions and millions of data points, you have to stat thinking outside of the box. Make sure you design the database well, because you will have to be able to change it as you learn more. The better the design the easier it is to change..and it will run faster. Spend a little time reading fundamentals of DB design or get someone to help you. A little more time perfecting the design will save you a massive amount of time later on. PHP and Javascript for the front end. This allows you to interact with the database in the manner in which you want to. Making it semi user friendly. Perl for the actual injection/feeder portion. I used perl because it doesnt have the random limitations that php has, ie timeouts and other random stuff they put in for it to be more useful for the web. In order to feed at rates that make sense, you will have to be creative. Use handlers that call other scripts so they work simultaneously, mail merge, and other things where you will have to think out of the box. But its fun. Good luck and feel free to hit my up if you have any questions.
Well thanks for the great tipps. CentOS really? I hava heared and read the opposite but I don't have knowledge to argue against CentOS. I am currently looking into Java since it seems to be one of the more popular choices for programming webapps. Also I found a decent video tutorial that gives me the basics. Not tha I am not valueing your advice ndcoder, but at least the feeding could be done with Java, too. Talking to programmers has showed me that more or less everything can be done with anything depending on your level of skill in that language. Are there any obvious disadvantages to programming the feeding part with Java? I know Perl is supposidly more beginner friendly but I have also heared it's a lot slower and I'll be having problems to think inside the box for a while. Also it seems with Java I even could do the php(not sure about the tracking) part with Java (not pretty looking though). So is there any reason to say no don't learn Java?
KillahBee, ndocoder took the time to reply to your question with what I think are good suggestions and you kind of completely dismissed them.
Hey I did rep him and he opened my eyes to a lot of things. I am a noob and thus was expecting a reply like "learn php and your good to go". Lots of love to your suggestions ndocoder I really appreciate them. Didn't want to sound like I dismissed you at all sry about that. The reason I mentioned Java is more the good free tutorial I found but also the fact that I could focus on it and mysql and get the project done completly with these two languages rather than learning 2 more. Also as far as I have learned up to now javas out of the box performance is supposidly a lot quicker which I hope would spare me of out of the box thinking on the feeding part. If he says fuck java for a feeding application it's inefficient and you can do it in the fraction of the time with perl and you won't get around php and javascript. I will do as he says. Didn't want to come across as unappreciative I noted his suggestion to do it as the optimum an advanced programmer who feels comfortable in those languages would do it. My question merely was could I go the route I outlined even if not the optimum in regard to looks, etc. but maybe the smarter way to go as a programming noob that is trying to get more comfortable with 2 languages rather than 4?
You can do it all in PHP. The issues with PHP timing out are irrelevant if you do it right...and to do it right you would use PHP scripts executed from the CLI rather than through the apache API or whatever your particular HTTP server is. The basic idea is that you would set what you want to mail within your web-based interface, and then have a cron job execute a CLI PHP script to do the actual injection. You could also use them to process bounces. Java is not going to be much quicker in this application because MySQL is going to be the bottleneck...that plus your actual mail servers' ability to queue mail. Also, Java applications execute on the client side and that means you cannot "set it and forget it" because you'd need to leave your browser open the entire time you're mailing...which is lame. Going the JSP route wouldn't alleviate the client-side execution thing but would then require additional crap set up on your server to process JSP.
Java apps can run server side... last i checked just about any device can run a jvm.. Even one of the "enterprise" level powermta management systems uses server side java to inject/manage crap.. I developed a list manager in php using mysql. If you structure your tables and queries properly mysql will not be the bottleneck. The system is currently handling ~50m records [yahoo/hotmail/cable/dsl doms], can inject over 400k/minute from mysql to a remote powermta instance using mailmerge.. @killahbee, if you have no programming experience, i would spend less time debating the performance of different operating systems and languages and more time learning basic programming concepts..
@DoldGigga Thanks I wasn't aware that java normally runs on the Client side and for the tipp on the PHP CLI. I will definitley look into to that. @nickphx Wow that is defintley good performance and its nice to see it can be done with php and mysql only. Btw I am not sitting around and debating here about what programming language to learn I am in the process but you are right. The biggest question mark I have in my head with this is the changing all domain names to my sending domain. I am not talking about the from, reply to and or return path, etc. This I can do easily but especially the links and creatives. Up to now this was all a manual process for me and very very time consuming. I honestly created thousands of php redirects renamed so they are unique and uploaded via ftp when using a pretty standard interspire feeder. This same question mark goes for things like tracking and maybe even unsubs. The long-term way it should be is select previously uploaded creatives with the click throughs in the feeder application and add select the sending server. How are unique click through links made efficiently? Am in the ball park using a catch-all php redirect or some .htaccess stuff and just having my feeder create unique links pointing to the catch all php redirect? Are maybe all of my problems concerning unique image URLs, tracking pixel same as sending domain solved with php catch-alls and redirect on the backend site? Sry for this unstructured rambling. I more or less got the catch all doamin idea have way through. Thanks for all the help.
Yes, you can run java programs on any platform as long as you install the appropriate JRE...but this guy is talking about a web-based injector app. Using java here would require it to execute client side or he'd need to set up something like tomcat to run JSP (java server pages). And it would be even more than 400K/min if you took mysql out of the equation and did something like dump the messages directly into the MTA sending queue. Mysql is a bottleneck because it requires more I/Os to perform operations that could be accomplished in a relative handful of direct filesystem I/Os...the benefit of using mysql is that it keeps things organized and lets you keep track of your campaigns. He could always hire someone but I agree that he should at least know some basics so he doesn't get taken for a ride. Your sending domain would be defined in your registrar and DNS records. The MTA can be configured to associate an IP with a domain name for "helo" purposes. If you're looking for real flexibility with your MTA you should check out Exim. It's what I use for some of my setups and it's pro. This is all stuff your injector program would do as it prepares each message. You would need to associate a URL with an ID and you can assign a unique ID to each email in your database, then the injector program would include both the URL ID and the email ID in the link so you know who clicked the link. I would use htaccess to rewrite the URLs and make them look "slick" but that's mainly for aesthetic purposes. Everything that would need to be unique per-message would be handled by your injector. The MTA just sends what your injector gives it. Or if you are like me, you have an MTA that is fully done in PHP and combines everything into one mailing system to rule them all.
that was accomplished using mysql feed from php cli. the most cpu intensive part i found with that injector was the hashing of the urls to make them unique for each user and some image generation and attachment i was doing . sending those off to a gearman cluster fixed that problem. i encode lots of data in each url, email, subject id, ip that sent mail, date, campaign id, randomize the url, then i encode it with a simple encryption algorithm. using htaccess i pass it off to a php script that decodes the url, grabs the data , logs the action type open/click/remove, etc.
Never said they were wrong about anything but there's always more than one way to approach a problem.
There are plenty of ways to approach any problem. I just happen to agree with their interpretation and approach and assessment down to a very surprising (to me) level of detail. It rarely happens.
I agree that ndocoder and Nickphx are spot on... @killahbee My advice is to take sometime and map out each component and tackle each piece one by one. All these items together may seem like a huge feat, but breaking them down may allow you to tackle a project like this... @Nickphx I'm curious about your DB design... Sounds like you're getting excellent performance from mysql. And your pMTA stats are sick!
it doesn't hurt that my database is on a 4x4core box w/96gb of ram with the db on an array of fusion io ssd drives..
I dont like/trust cron (well not that serious, i do use cron for other things), so I just have a perl script that runs on an infinite loop. Solves the issue of the cron calling the same script before that script finished and a few other small issues as well. Plus, I'm working on wrapping all of it up into a neat little daemon. So many ways to skin this cat.
i've had less success trying to get perl to run as a 'daemon'.. maybe i'm just a sloppy coder. you can use a "lock" file to stop cron from running same thing before it's done.. just have your script delete the lock file after your script is done running. i use this for perl: for php it's more elaborate.. then i just require_once("lock.php"); Process_Manage::lock(); at top of script and when script is done Process_Manage::unlock();
I'd always recommend a lock file solution with regularly re-launched scripts over a single script that is supposed to run indefinitely. The longer a script runs the greater chance it will accumulate memory leaks. Better to re-launch early and often.
