Who is this “Ollie Parsley”??

Developers: Cache your RSS

May 16th, 2010. Published under Development. 1 Comment.

Background

First of all I think it is important to note that I have never received any training when it comes to PHP, MySQL or Apache. I have just finished a degree course but that was mainly Java.

This post is done from a web services point of view, not a developer reading an RSS feed . Anyway! I have recently moved servers to Linode, which I would highly recommend if you know your way around Linux. I moved all my small sites like this blog, Hootware, H1 Debate, Love Hate Marmite etc onto the new box and everything was running fine as they don’t have much traffic and on every blog I host I use WP-Supercache.

The problem

But when I moved FootyTweets over and got it all running I noticed that after about 10mins apache had ground to a hault and clogged up the server. It took me a good 20mins to figure out what had been going on. I didn’t realise it but almost every second there are people out there that refresh about 20 RSS feeds from FootyTweets. As I had never anticipated many people would use the RSS feeds I never thought to cache them. This was baaad news!

Every time an RSS feed was requested it went to the database, requested x recent news items and made an RSS feed from them. Now the news feeds only get checked every half an hour, so after the first feed requested in that 30 minute slot I was simply wasting database connections and processing time doing the same SQL statement.

MY solution

This is when I jumped into action. I know there are different ways of caching but my favourite way is a bit similar to WP-Supercache, which is simply save a static copy of the RSS feed so a file when the first request is made. Then all subsequent requests simply check to see if a cached version exists, if it doesn’t it will create the file, otherwise it will simply server up the static file. Then whenever a news item comes in the RSS feed for that team is deleted ready t be recreated.

And the people rejoiced

As soon as I put this back into action the server was back to normal and apache seemed to appreciate the effort I went into to create the cache.

As I’d done this type of caching of files before it only took me about 10-15mins to implement, but the first time it took a bit of figuring out.

Just as I said before, this is not the only way there are other services that can do the same thing. Everything that was written above was my own opinion .