Performance Issues
Well, I'm sure that some of you may have noticed that for the last five days or so our website has been extremely sluggish. Wow, it's only been five days? Feels like it was a month. Anyway, for some reason our website started to take over a minute to load one day, and that was strange.
Looking into it a bit more it looked like it might have been persistent database connections causing the problem. This happens when you create "persistent" connections that don't close at the end of a script. Rather, they stay open until they time out, which can quickly cause the number of connections to max out. At that point you will either get an error message that says "too many connections", or there will be a REALLY long lag while the server waits for connections to time out before establishing new ones. Seemed like a good, easy explanation, right? Wrong, there are no persistent connections in the PHP code. Since I built the site I was confident that this was not it, and it wasn't.
I then started to worry that it was the actually programming of the site itself. We are actually optimizing the programming of the site anyway, just to make sure that it never is the programming. This theory held until yesterday, when I realized that it was also not my programming. Whewwww!!! I was worried there. The only other option was that it had something to do with our hosting server. There were a couple of pieces of evidence pointing this way, in hindsight.
The first is that our problems coincidentally started expressing themselves right around the time that our hosting company had been "nudging" us to upgrade our servers. It was a free upgrade, as they have recently add improved services, but we needed to actually migrate over to them. Looking back, that might have been a clear signal, and it wasn't until a few days ago that I noticed (despite what the support people were telling me) that the server's CPU and overall load was much too high. Put this together with the fact that our problems just "appeared" one day without any other event or provocation and I should have known.
I'd say the final piece of evidence comes from the fact that once we did the migration (which may be the topic of an upcoming "Customer Service" post) the issues have been solved. The site is back to loading quickly, the server load is back to a normal range, and everything appears to be working fine. A few little glitches are going to present themselves due to the upgraded version of PHP (and updated config settings), but I'll take care of them as the appear.
Knock on wood, as it seems that every time I say "everything is working great" the next day some bomb drops. However, I'm glad that we got through this one, I apologize for the inconvenience that any of our users experienced, and excited to get our programming all polished up.
This post is filed under Developers' Corner and has the following keyword tags: database, mysql, performance.
