How Google Routes Around Outages

No, that’s not my title.  The folks over at Data Center Knowledge have an interview with Urs Holzle, Google’s Datacenter Operations Manager:

Making changes to Google’s search infrastructure is akin to “changing the tires on a car while you’re going at 60 down the freeway,” according Urs Holzle, who oversees the company’s massive data center operations. Google updates its software and systems on an ongoing basis, usually without incident. But not always. On Feb. 24 a bug in the software that manages the location of Google’s data triggered an outage in Gmail, the widely-used webmail component of Google.

Just a few days earlier, Google’s services remained online during a power outage at a third-party data center near Atlanta where Google hosts some of its many servers. Google doesn’t discuss operations of specific data centers. But Holzle, the company’s Senior Vice President of Operations and a Google Fellow, provided an overview of how Google has engineered its system to manage hardware failures and software bugs.

Read more at Data Center Knowledge: How Google Routes Around Outages.

Say Thanks

I was surfing around the blogosphere today when I came across the Manager Tools Blog.  Now, I’m not a manager (nor do I think I’m qualified or would want to be at this point in my life), but I found this blog a few weeks ago on Rickey’s (my manager) blog.  Every now and then I’ll browse through their stuff and ran across something this afternoon that I think applies to (some of) us in the IT world.  The post is titled “My Postman is Lonely.”  No, I’m not in the postal business (my patience could be described as “postal,” though).  But, the point of the post has to do with the service industry and how too often people from any “service” industry are seen as a service instead of as a person:

There are lots of people with jobs like his, where you meet lots of people but only briefly, and you’re seen as a service and not a person. Receptionists, security guards, catering staff, the man who fixes the heating and the guy who refurbishes your computer. I think they are lonely too sometimes.

It’s not a long post by any means, but I can remember not too long ago being a “service.”  Luckily, I work in a place now where people appreciate other people and work/accomplishments are rewarded.  And, by reward, I don’t necessarily mean a monetary reward – a simple “good job” or “atta’boy” goes a long way in a lot of cases.  Remember that.

Error 0x800704C8 in VMM

I was trying to deploy a new virtual machine today via Virtual Machine Manager, and kept getting this error (names changed to protect the innocent):

Error (12700)
VMM cannot complete the Hyper-V operation on the [virtualhost.domain.com] server because of the error: ‘[VIRTUALGUEST]‘ failed to add device ‘Microsoft Emulated Ethernet Port’. (Virtual machine ID 26DD8B56-0594-446A-8084-405218737EB0)

The Virtual Machines configuration 26DD8B56-0594-446A-8084-405218737EB0 at ‘J:\VS\[VIRTUALGUEST]‘ is no longer accessible: The requested operation cannot be performed on a file with a user-mapped section open. (0x800704C8)
(Unknown error (0×8000))

Recommended Action
Resolve the issue in Hyper-V and then try the operation again.

Well, that’s really good advice – clear the error and try again.  So, a quick Google-ing turned up a Microsoft KB article: Creating or starting a Hyper-V virtual machine on Windows Server 2008 or Microsoft Hyper-V Server 2008 may fail with error: 0x800704C8.  Oddly enough, this was exactly my issue:

CAUSE

This issue can be caused by antivirus software that is installed in the parent partition and the real-time scanning component is configured to monitor the Hyper-V virtual machine files.

RESOLUTION

To resolve this issue, configure the real-time scanning component within the antivirus software to exclude the following directories and files:

  • Default virtual machine configuration directory (C:\ProgramData\Microsoft\Windows\Hyper-V)
  • Custom virtual machine configuration directories
  • Default virtual hard disk directory (C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks)
  • Custom virtual hard disk directories
  • Snapshot directories
  • Vmms.exe
  • Vmwp.exe

Notes:

  • If virtual machines are missing from the Hyper-V Management console, you must configure the antivirus exclusions, and then restart the Hyper-V Virtual Machine Management service.
  • If the error code was 0x800704C8, it is likely that the virtual machine configuration file was corrupted and the virtual machine may need to be re-created or restored from backup if restarting the Hyper-V Virtual Machine Management service does not resolve the issue.

Chalk one up for Microsoft KB articles.  Now, a little further down in the KB article, it suggests (per Planning for Hyper-V Security) installing the antivirus on the guest (which we do) and not on the host (which we do, too):

Do not run any applications in the parent partition. Run all applications on virtual machines, which use child partitions. For example, if antivirus is required, be sure to run it on the VMs rather than the parent partition. Keeping the parent partition free of applications and running on a Windows Server 2008 core installation means fewer host updates, since nothing needs software updates except the Windows Server 2008 core installation, the Hyper-V service components, and the small (~600KB) hypervisor.

Nigerian Money Scam

Here’s an interesting (and funny) post from Mike Nash (of Online-Armor) on the old “Nigerian Money Scam.”  Remember, sending money to strangers (especially strangers in another country) is bad.

The post.

And an excerpt:

So, I’m sitting there today working on something for a client when I received an unsolicted Skype Message with an “Important Business Proposal”.

I normally mess with these guys a little, just to waste their time , but as I was on the phone to a client I just decided to get rid of him quickly.

As you can see – he was suprisingly honest about his ultimate intentions.

Windows Web Application Gallery

I stumbled across something today I think is pretty cool: the Windows Web Application Gallery.  From iis.net:

The Windows Web Application Gallery makes it easy to explore, discover and install popular community ASP.Net and PHP applications on Windows. Users can browse and view applications for different types of Web sites, ranging from photo galleries to blogs to ecommerce sites.

The Web Application Gallery integrates with the Web Platform Installer 2.0, so that when a user clicks “Install” for an application, the Web Platform Installer 2.0 launches with context set on the user’s selection. This powerful combination of application discovery and simplified installation makes it easier than ever for community-based applications to run on Windows.

This has some neat implications.  For one, developers can have their stuff put out for the whole world to see – and install in a few clicks.  Another, if you run a web server (like me) that has several blogs on it, this could streamline installing, say, WordPress.  Or Gallery for your users.

I don’t get terribly excited about some stuff Microsoft does, but this is pretty cool.  That is, if it takes off.

Self Signed SSL Certificates

I’ve been trying to figure out how to get an SSL certificate on the cheap.  I mean, 300+ dollars per year is a little ridiculous to get an encrypted pipe between client and server.  You see, I don’t really care if my identity is confirmed.  I know what server I’m connecting to (that is, unless you hack me because I just told you that).

Anyway, there is an open source project called (go figure) OpenSSL.  OpenSSL is an SSL toolkit – that happens to be free for commercial and non-commercial uses.  While they offer a self signing feature, it’s not generally recommended for production applications (you’ve been warned).  For me, however, it works perfectly.  And there’s even a version for Windows.  Now, the cool thing about this is it will create a private key in a variety of encryption protocols, it will create your certificate request (if you’re well-to-do and can afford a real one), and it will even create a “test” certificate (which is what I wanted).

So, I downloaded the 64-bit version for Windows.  When you’re done (use the defaults – they work just fine), open a command prompt, browse to your OpenSSL installation folder (C:\OpenSSL be default) and type:

openssl

You’ll be at a prompt that looks like “OpenSSL>”.  Once you’re here, you’re ready to create your private key and your certificate.  In my case, the following command works perfectly.  It creates a private key and a certificate all at once.  If you’re going to buy a certificate, you’ll need to Google the procedure for creating a private key and a certificate request, then do whatever you need to do to install the certificate from that company into your program(s) that require it.  But, I digress – the command:

OpenSSL> req -new -newkey rsa:1024 -days [DAYS] -nodes -x509 -keyout www.example.com.pem -out www.example.com.pem

You’ll need to follow the following prompts (asking for your country, name, email, etc.).  The above command will create a 1024-bit RSA private key, then create a certificate (I think it does the certificate request in the background).  The certificate will expire in the number of days you set via the [DAYS] variable.  The -keyout tells OpenSSL to export your private key to a file called “www.example.com.pem” and the -out switch tells it to export your certificate to “www.example.com.pem.”  Now, I’ll state the obvious and recommend that you change the file names (ie, examplekey.com.pem and examplecert.com.pem).  The key and certificate are both created in the same folder (C:\OpenSSL by default).

Now, the one caveat I will offer is this: this self-signed certificate is not trusted by anyone.  Internet Explorer, Firefox, Outlook, Safari, etc. will all reject this certificate as “identity unknown.”  For me, that’s okay.  For someone else, think about it first.  You can get around the constant “this certificate is not valid” prompts by adding the certificate to the Trusted Root Certification Authorities in Windows’s Certificates Manager.

Now you know.

ActivityTimeout

I’ve been trying to get another website I manage to allow me to upload files (usually around 40MB) via the browser so I don’t have to open up FTP.  The problem, I’ve been having is something has been going wrong and the upload fails.  Naturally, no real or useful error message is given.

Here’s my setup: IIS7, PHP 5 (using FastCGI).  When I would click the “Upload” button from the webpage, things would chug along and eventually fail.  The only thing the browser would say is that the connection had been reset – which tells me nothing.  Did my ISP have a hiccup?  Did the server shutdown?  Pretty useful, eh?

After trying everything I could find on Google, I tried the simple thing: I changed the ActivityTimeout under FastCGI Settings.  This is defaulted to 120.  Basically, this says that if the php-cgi.exe process doesn’t talk to IIS for two minutes, timeout.  Well, while the PHP process is processing my upload (the verb IIS uses is “POST”), it seems that it doesn’t talk to IIS, so IIS times it out.  Well, you try uploading a 40MB file from some DSL site in the world and see if you can do it in two minutes.  I know I can’t.  I changed the timeout to 600 seconds, and now I’m happily uploading files.

I suppose this could be dangerous if PHP hangs and the process gets stuck.  It might make users wait 10 minutes if something screws up.  I suppose I’ll tackle that issue if it comes up.

Wassup

So, it seems (after much trial and error) that one of my plugins was the cause of my speed issues.  Wassup is a statistics plugin, and was causing major performance issues.

I like Wassup because it provides good data, but I’m sad to say that I have deleted it.  Now, you can enjoy my blog without waiting 15 minutes for a post to load.

Speedup WordPress on IIS 7.0

Here is a nifty article on speeding up WordPress.  I’m still not terribly thrilled with the performace on IIS.  I’m not sure if it’s the PHP Engine or the database backend now.  I’ve noticed that php-cgi.exe spikes at 100% when viewing some pages.

Anyway, here it is.

It gives two options for “speeding” things up: IIS’s Output Caching and WP Super Cache.  Each have their downsides, but maybe one will work for you (and me).

Disable Hibernation

If you’re in need of disabling hibernation on your 2008 servers (or your Vista desktop, as the case may be), open a command line and type this:

powercfg.exe /hibernate off

To turn it back on, replace “off” with “on”.

This will do away with that annoying file (hiberfil.sys) at the root of C:\ (or whatever your system volume’s drive letter may be) that is the size of your physical RAM – thus freeing up much space.

Keep in mind, however, that hibernation cannot be turned on if you have Hyper-V installed.  I’m curious, though, why would you hibernate a server?  And before those of you that use 2008 as a workstation complain, why would hibernation be enabled by default?