Sometimes it’s not so Good when People are Happy to see You

“There he is!” both my fellow engineers said as I walked into the office this morning. I knew right away that this was not going to be the Monday I expected it to be.

“The database is down,” my boss said before I reached my chair. “Totally dead.”

“Ah, shit,” I replied, dropping my backpack and logging in. I typed a few commands so my tools were pointing at the right place, then started to check the database cluster.

It looked fine, humming away quietly to itself.

I checked the services that allow our software running elsewhere to connect to the database. All the stuff we controlled looked perfectly normal. That wasn’t a surprise; it would take active intervention to break them.

Yet, from the outside, those services were not responding. Something was definitely wrong, but it was not something our group had any control over.

As I was poking and prodding at the system, my boss was speaking behind me. “Maybe you could document some troubleshooting tips?” he asked. “I got as far as the init command, then I had no idea what to do.”

“I can write down some basics,” I said, too wrapped up in my own troubleshooting world to be polite, “but it’s going to assume you have a basic working knowledge of the tools.” I’d been hinting for a while that maybe he should get up to speed on the stuff. Fortunately my boss finds politeness to be inferior to directness. He’s an engineer.

After several minutes confirming that there was nothing I could do, I sent an urgent email to the list where the keepers of the infrastructure communicate. “All our systems are broken!” I said.

Someone else jumped in with more info, including the fact that he had detected the problem long before, while I was commuting. Apparently it had not occurred to him to notify anyone.

The problem affected a lot of people. There was much hurt. I can’t believe that while I was traveling to work and the system went to hell, no one else had bothered to mention it in the regular communication channels, from either the consumer or provider side.

After a while, things worked again, then stopped working, and finally started working for realsies. Eight hours after the problem started, and seven hours after it was formally recognized, the “it’s fixed” message came out, but by then we had been operating normally for several hours.

By moving our stuff to systems run by others, we made an assumption that those others are experts at running systems, and they could run things well enough that we could turn our own efforts into making new services. It’s an economically parsimonious idea.

But those systems have to work. When they don’t, I’m the one that gets the stink-eye in my department. Or the all-too-happy greeting.


CloudFlare = Awesome

So by now you’ve probably heard of “the cloud”, but you might be vague on what the cloud actually is. That’s OK, the cloud is by nature vague. In short, it’s just a name that applies to what the Internet has been trying to do for a long time: information without location. You put a photo up in the cloud, and it’s just “out there”, not on any particular server, not in any particular data center, not in any given country. Could be there are copies of it all over the place, and when someone wants to look at the picture, The Cloud serves up the copy closest (in Internet miles) to the person who wants to see it.

This requires a lot of expensive equipment. Google and Amazon are the biggies in the cloud, but there are others as well, who, for a price, will host your data in a ‘cloudy’ way. In return, people around the world can load your stuff faster.

This humble blog is in the cloud. When you load a page here, roughly half the time the request doesn’t even reach my server (protected in a bunker somewhere in Nevada), but is instead served up from one of CloudFlare’s data centers around the globe. It’s pretty sweet, and has reduced the strain on my server (not that it’s working that hard anyway) while improving the Muddled Experience. The cost for this service? Nothing. It’s free.

I totally win.

CloudFlare also blocks a few hundred spammers each week, before my server has to go to the trouble of blocking them. They compile usage stats and provide other interesting information, and cut the load time for the blog about in half.

They’re a friendly bunch, too; when I suggested upgrades to their interface they wrote back with specific questions as well as thanks. A site they hosted was attacked from China a while back, and it brought down part of their network. They were right up front about the issue and what they were doing about it, and advised people on how to ‘de-cloud’ until the crisis was over. Not everyone was happy, but I was impressed. Soon after reading those communications I signed up.

How can they offer something like this for free? It’s the upsell, of course; they offer premium services. In addition they create a platform for other companies to sell stuff to me. Some of those services are pretty cool, too, though I haven’t dipped my toe in those waters yet (for instance, there’s a free service that checks your site now and then to see if it’s been hacked).

Overall, I can’t think of any reason NOT to use CloudFlare. Check ’em out and tell them Jerry sent you!