RSS Feeds and Speed

For those not technically minded, here is the quick summary:

The RSS feed feature for NovelRank is used across more than 9000 feeds each day, even for books that are no longer being watched by the person who added them. However, they still require the website resources to provide them. An improvement today will ensure that this invisible resource hog will not slow down NovelRank’s server in the future.

Okay, if you are still with me, here is a more specific rundown. Before User Pages and the User RSS feeds came along, there were separate RSS feeds for every book (ASIN), and for each book there were up to 8 different feeds (sales-1 and rank-7)! People dropped these into their reader or home page, and while I’m sure many still use them, many more have been long since forgotten. However, the computers and servers that check them continue to check for updates, multiple times per hour, even if the book is INACTIVE! You can imagine the load this has on the servers.
Previously, there was a simple cache in place to not ‘refresh’ the content no more than once per hour, just like the rest of NovelRank. However, it still required connecting to the web server (Apache) and activating PHP’s engine. This is resource intensive, to say the least. It took all day today (seriously, I didn’t have breakfast till 1pm), but I’ve worked out a new methodology that should improve this experience. Now, using Apache’s mod_rewrite module, the cache will be served up. If the book is active, then every hour, when there may be new data, it will refresh the cache upon request. If the book is inactive, the cache will only refresh once per day. That removes a hell of a lot of server resource overhead. Finally, I’ve also begun serving a 410 (Gone) server error for RSS feeds on books that are inactive on all domains (sales feed) or for that particular domain (rank feed), to stop the zombie requests from coming in the first place. This also means that if your book is inactive on a particular domain, you will no longer see a rank RSS feed link for that particular domain. This is only for RSS feeds on the book details page (not the user RSS feeds).
I’ll be continuing to monitor the server for other improvements in an effort to continue to serve a fast, responsive web experience for the growing NovelRank audience.


If you read all of this, I salute you.

Similar Posts:

Be Sociable, Share!