IT Operations Shouldn’t Be A Big Data Problem

by August 15th, 2012

The IT Management Arms Race (to Nowhere) Continues in Full Swing…

VMware is acquiring yet another piece-part for its management portfolio: Log Insight. As many have noted (here and here), this is the latest addition in a long list of acquired tools for VMware (see sidebar).  Will this bring VMware any closer to delivering a useful operations management solution? Does it set the proper foundation to address the management challenges in today’s virtual data centers (or, software defined data centers, as some would call them)?

The fact is that, data centers are increasingly comprised of more technologies and moving parts, with complex dependencies—vertically across the IT stack and horizontally across multiple clusters and clouds. Networks, storage, servers, virtual machines, applications, ADCs, and more are all interrelated in complex, dynamic, continuously changing dependencies—and together must deliver the services required by the business. Even in moderately-sized environments, there are literally tens of thousands of moving parts that need to be managed, architected, provisioned, configured, monitored, controlled, orchestrated, alerted, remediated, reported on, etc.

The Sins of the Past – Revisited

In the last two decades, developments in the IT management software space, led by the “Big Four” framework vendors, would be characterized as a race to nowhere. For each of the technologies and functions above, a point tool was developed (most likely by a startup), and the Big Four acquired, packaged, and repackaged them. They cobbled them together under a “single pane of glass” to hide the lack of underlying integration, and marketed them as the answer to all of our problems. Instead of solving the management problem, they created a management nightmare!  Moreover, most of these tools focus on collecting data—a lot of detailed data—that could be used to alert administrators and generate reports.  Not only does the collected data not solve anything, but it creates a whole new problem: a Big Data problem.  In addition to managing all of the components in our data centers and clouds, we now have to store, manage and manipulate the huge volume of data generated by the so-called IT management solution, and figure out how to extract meaningful information from it.

So, here we are with the acquisition that triggered my thoughts—essentially we have a vendor acquiring technology to try and help their customers make sense (and manage) the reams of data they are providing them with. Where does this end? It’s a bottom-up approach and we (as an industry) will never get there on that path. Why VMware has opted to recreate the “sins of the past” with their management strategy is a healthy discussion—for another time. The point is that the trajectory for IT management software needs to change—and virtualization and the “software defined data center” is the perfect inflection point to make the shift. As an industry we need to fuel the change—not support the status quo. Unfortunately, VMware has not led in this regard.

Constrained by Brittle Intelligence

IT operations management based on a collection of limited function and nonintegrated tools lacks the agility, automation and intelligence required to control today’s dynamic data centers and clouds. Collecting data, filtering it to make it more manageable, and presenting it in a dashboard is nice, but it’s not prescriptive. The “heavy lifting” of analyzing, interpreting, and taking action on the information requires humans in every step of the management loop. Up to now, it’s been tolerable for many; however, virtualization, cloud and (now the new buzzword, software-defined data centers) are changing the game and fundamentally breaking traditional approaches. Complexity, interdependencies, scale, and the pace of change are forcing a different approach to IT operations. Further, virtual and cloud infrastructure enables new (and better) management approaches. So, how long will we continue to be restrained by brittle intelligence?

Virtualization broke down the silos of the IT stack—the interfaces between the different layers are no longer what they used to be. We can no longer continue to manage networks, storage, servers, applications, etc. separately. We can no longer have separate tools to monitor and report on servers, storage, networks and applications. We can no longer use disparate tools for capacity planning, workload placement, performance management, on boarding, workload provisioning, workload configuration, and balancing memory and CPUs across hosts. These functions all deal with allocating resources most efficiently to meet application workload demand. This issue must—and now can—be addressed holistically by an integrated solution. Furthermore, we must shift our focus from collecting, alerting and reporting to controlling, preventing and optimizing. Instead of alerting and reporting on degradations and exceptions we must focus on controlling and maintaining the environment in a “healthy” state, preventing degradations and exceptions and aligning the IT infrastructure to meet business goals. This is a top-down approach, not a bottom-up data collection exercise.

Unfortunately, VMware’s vision of IT Operations Management lacks innovation.  The company’s latest acquisition is very telling about the problem they have created for the industry—and themselves.  The Log Insight technology is a Big Data band-aid to the issues their management architecture(s) face with the scale, velocity, and complexity of dynamic data center and cloud environments today. While Big Data is a hot topic in other use cases, there is nothing cool about this intersection of Cloud and Big Data.