Some techy stuff about how I migrated Dabr to it’s new home last month, cross posted from one of my other blogs:
After a recent purchase of VPS hosting from WebFusion, I wanted to move my high-traffic site dabr.co.uk onto it. With the site exceeding one million daily page views, I really had to be careful about how I’d move the site to it’s new home.
Using the “Hostnames” report within Google Analytics, I was able to identify that 10% of users access the site via the m subdomain and 40% use www – the rest not using any subdomain at all. This was a real stroke of luck as it allowed me to test the new server with a controllable percentage of the total traffic, simply by updating the DNS records.
When given 10% load the VPS handled just fine, so after two days I moved on and tried it with 50%.
A second report from Google Analytics also helped tremendously: hourly visits. With about 45% of hits coming from Indonesia (yes, really), I needed to know what time was the best window to try any drastic changes to the servers. The graph clearly shows that the longest quiet period I get is between 6pm and 9pm GMT – when most of Indonesia is asleep and web traffic drops in half:
On the first few occasions that I did try 50% of the load pointing at the new server – it buckled and almost completely fell over. A chat with Nick, a friend from Heavy Sumo, led me to explore the Apache configuration – and what a bizarre experience that was.
Here’s the settings that were used by default on the server, compared with the Apache defaults and the settings on the old server:
Setting WebFusion VPS Apache default Old server StartServers 1 5 8 MinSpareServers 1 5 5 MaxSpareServers 5 10 20 MaxClients 10 256 256 MaxRequestsPerChild 0 1000 4000
Notice the difference?
I have no idea why WebFusion would have their defaults so low. Changing these to match the settings of the old server had a significant boost on performance.
I left 50% traffic on the new server for a couple of weeks, and tried a couple of times to add a bit more, but users quickly complained about the site slowing down. The slow down matched up to the Indonesian peak time around 2pm GMT, so I rolled back the DNS to point to the old server again.
This stage required a little further exploration of Apache performance tuning articles. One interesting fact I discovered was that Apache has a hard-coded MaxClients limit of 256, which you can only raise by recompiling Apache – so that wasn’t particularly an ideal option.
The Apache Performance Tuning article from devside.net was the answer to my problems. It suggested a number of different solutions, two of which were to adjust Keep Alive and Timeout settings.
Setting Webfusion VPS Devside Timeout 300 45 MaxKeepAliveRequests 100 80 KeepAliveTimeout 15 2
These changes, combined with the forking settings above, have made the server run much more smoothly. In fact, it’s now taking 100% of the traffic and people are noticing an improvement in speed.
With the new server running well for a month, I then asked the previous host to delete the old server. Hopefully the new server should last a while before I need to consider load balancing to a second server.