Turbo Talk

In October 2009, Gartner estimated that only 16% of workloads worldwide are running in virtual machines, although tremendous growth is expected in the coming years.  Not surprisingly and roughly in line with Gartner’s estimates, most customers tend to be about 20% or 30% virtualized, with ambitious plans for growth in the coming year.

However, some organizations are outliers.  They have virtualized the majority of their IT environment and are seeing benefits above and beyond the typical server consolidation and disaster recovery use cases.

I like to call these folks the uber-virtualized, and in this post I’ll discuss some of the best practices we’ve learned from them!

1)  Pay attention to your storage environment, because there is a good chance it’s where the bottleneck lives!

The fear of storage bottlenecks keeps the uber-virtualized up at night.  When you’ve virtualized most of your IT environment, that is going to cause additional stress on your SAN because of all the virtual disks you’re storing there.  Rather than throwing storage capacity at the problem (at additional cost), much time and effort goes into poring over storage array and VirtualCenter data, trying to find optimization opportunities.

There are certainly some monitoring tools on the market which can aid in this process by gathering numerous bits of utilization and performance data from hosts and storage arrays.  However, these tools (just as VirtualCenter) leave the administrator to make the final decision about how to rebalance the environment and mitigate the risk of storage bottlenecks.  Fortunately, the uber-virtualized have been working with VMware technology for many years, and are often able to make the right decision based on their experience.

VMware has also recognized that storage challenges can really hurt virtualized IT deployments, and have responded by developing new technology like IO DRS.  This is a good first step, although the VMware administrator will need to have the experience to recognize the proper thresholds to configure to trigger a migration.  With 10 or even 100 VMs, this is fairly simple to do.  However, with hundreds or thousands of VMs in an environment that’s tightly managed (50%+ utilization), deeper analysis needs to be done looking at all resources together (CPU, memory, and I/O) before making workload balancing decisions.  Otherwise you’re risking performance problems and downtime.

2) Are you (CPU) Ready?

CPU Ready is one of the key parameters that the most experienced VMware administrators examine when they see a performance problem.  In fact, it is often the first thing they’ll check when debugging.  Learn to love this statistic and what it means, because it can help you identify virtual machines that may be oversized and that cause your environment to perform poorly.

(By the way, here is a nice Powershell script that will grab CPU Ready stats for all of your VMs!)

3) Automate lightly, young Padawan.

One thing that surprised me in talking to the uber-virtualized is that some are skeptical about the use of automation like DRS in their environments.   I thought everyone would be running DRS in fully automated mode, and have their feet up on the couch at home while drinking a beer since their environment was managing itself   But there were some organizations that felt that DRS didn’t give them everything they needed (particularly in the IO department).  It will be interesting to see how IO DRS, once released, will address some of these concerns.

Are you one of the uber-virtualized?  Care to share any of your secrets, tips, or tricks?  Please feel free to leave them in the comments, your fellow VMware administrators and architects will appreciate it!

After spending most of the last 6 years working in the virtualization space as a vendor (now VMTurbo and previously VMware), it’s funny to see how much things change as well as how much they stay the same.

One thing that has definitely not changed is that it is still very hard to move from what I like to call the ‘1st wave’ of virtualization (test & dev systems or low criticality production systems) to the ‘next wave’ (e.g. business critical production systems, heavily utilized databases).

On that note, I wanted to share a few practical tips that I’ve picked up along the way that have helped customers and partners keep the virtualization momentum going – and can help keep your virtualization momentum going in 2010!

1.  Use a disaster recovery or business continuity project to spur additional virtualization and consolidation.

Most companies have challenges around meeting disaster recovery and business continuity goals.  DR in the physical server world is tedious, error-prone, and in my experience mostly ineffective.  If there is a DR initiative at your company, it is a good bet that some of the problems you are trying to solve could be addressed by virtualizing those systems which  don’t have DR capability today or that have been problematic to recover using traditional physical server techniques.  Another tactic that I really like and have seen a few times is using your test and dev environment for DR.  Most virtual server environments I see still have plenty of capacity with which to handle a burst in the event a DR scenario occurred, so having a hybrid test/dev/DR environment is a great way to leverage an investment you’ve already made.

2.  Use a hardware refresh as an opportunity to virtualize.

Most IT shops refresh their server hardware every few years.  Why not use the refresh as an opportunity to remove hardware from your datacenter while adding flexibility to your operation?  Some of these systems may represent some of the more challenging applications to virtualize, and you may receive some resistance from application owners who are new to virtualization, but the CAPEX (and potentially OPEX) savings will be hard to ignore.

3.  Educate your peers.

Many companies do ‘lunch and learns’ or other informal gatherings where the virtualization team leads will discuss how server virtualization works.  These gatherings are a great way to get your network, storage, and applications guys up to speed with your specific initiatives and virtualization technology, and get them talking and asking questions.  This education and relationship building will pay dividends when you start to move more critical applications into virtual machines and need to work closely with other groups within IT on capacity planning and troubleshooting.  Just ask the network guys, they’ve been getting blamed for years for problems that aren’t theirs!  Fortunately for them (and sadly for the virtualization administrator), the new whipping boy is the virtualization environment, and educating your peers can help mitigate this challenge.

4.   Connect with others in your city or industry who have successfully made it to the ‘next wave’ and gather best practices.

Certainly the web and social networking give us a great way to connect with virtualization experts, but there is still no substitute for face-to-face discussions or phone calls where you can ask questions directly to someone who has done it before.  If you know of another company in your industry or city who have already made it to the next wave of virtualization, and have learned the lessons (good and bad) along the way, reach out to them and see if they’d be open to a discussion.  I’d also recommend including when possible any key peers or managers in these calls and meetings.  This way, they have the opportunity to ask questions as well as internalize the information.

5. Measure and then publicize your success.

Don’t be afraid to let people in your organization know that you’ve saved money, increased responsiveness of IT to the business, and built a strategic, virtualized platform!  Keep an eye on your ‘before’ and ‘after’ metrics, and share them with management as well as folks on the business side.  Your results help build the confidence within your organization that you have a good handle on building and operating a virtualized environment, and are fully capable of onboarding additional applications and business units.

What did I miss?  Are there other techniques that have worked for you?  Please share them in the comments.