About one week ago I was notified that the department for which I work in the company who employs me would have to move all of their servers from one datacenter to another, immediately. This may not seem like an issue but it really presented a huge challenge to my group and I will explain why.
You see, this specific group is somewhat of an oddity within the company because the group uses a number of enterprise-level data management systems all with a highly-specific purpose, a relatively high value and use the systems worldwide. Unlike many groups within this company, my group (generally) maintains their own data-management systems. Therefore, the mandate to move is not specific to my group but our group has a relatively high impact to support ratio. Very specifically, I am the only engineer for this group, so it’s generally my responsibility to make all of this happen, smoothly.
After all of the dust settled, my group had a number of servers which needed to move. Altogether, we needed to move nine servers. Of these nine servers:
- 1 was a physical database server running RHEL w/ Oracle
- 1 was a large file physical server running Windows Server
- 7 were virtual servers running various flavors of Windows Server
The majority of Windows servers are running JSP or ASP web applications and all of the servers have the usual server-reated goodies, like CNAMEs, monitors and so-on.
These applications, as I mentioned, are generally used within my department but are used by relatively “high level” employees just about 24 hours per day. Therefore, I wanted to minimize the service disruption to my department for all services.
The whole point of writing this is so I can share some of my lessons and thoughts after the fact. In general, this whole experience was a lesson to just keep things simple. Technology is awesome but I think too many people get lost in complexity.
Virtualize when possible
Earlier this year I spent a good deal of time moving my group away from physical hosts to virtualized hosts. We have used more than one piece of virtualization software and we have virtual hosts/clients of vastly different sizes and capacities. I made the push to push our application and database into a virtualized space. For the most part, we are not running anything which a well-provisioned (4 CPU) VM cannot handle.
I can easily say that using virtual machines made moving the bulk of these servers relatively painless. Instead of worrying about (intensive) imaging and finding identical hardware or reinstalling on new hardware, we simply moved the virtual machines to new hosts and fired them up.
Put simply, when changing a bunch of things at once, virtualization was a huge win for the fact that the transition introduced a very small number of changes into the application environments.
Finally, I know people gripe about the power of VMs; if you have applications with need a lot of horsepower, well, push the vendor to support clustering.
Configure and build intelligently
When building or configuring an application, please, never:
- Use a static IP address unless you have a decent reason to do so
- Use a hostname which is likely to change (like a FQDN based on the datacenter of a host)
- Daisy-chain DNS records
Honestly, the only real issues which came up on my end fell into one of the three above. Thankfully, when we moved to virtualized hosts (for the most part) these issues came up so I had some very good ideas where to look when things did not work properly. Amusingly, vendors were the ones who went hard-coded-IP crazy while our internal people were, oddly, daisy-chaining DNS records.
Also, important and related to this point – if something goes wrong, do something very simple to isolate the likely cause. Just think – what has changed?
Computers have no black magic, I assure you. In any case, re-read the plug on why VMs are handy as far as limiting changes.
Communicate like a human being
Admittedly, I always get irked when I see engineers communicating (good) in ways which are generally useless to non-engineers. If you have customers who are impacted by a large-scale change, communicate in ways they will understand and, I promise, you will have less overhead dealing with follow-up questions.
For example, my company has engineers who regular issue change notices which are, for all intensive purposes, very detailed but amazingly cryptic and generally far too long for anyone to read via email.
Even complex and detailed systems information can be phrased into a three-paragraph email.
Instead of firing off technical details at customers, talk to them, intelligently but non-technologically. It is MUCH easier to include a “for more detailed and technical information, see this page” link than to just communicate on the wrong level to begin with.