3 Pillars Is Dead: Don’t Let DevOps Die With It

The truth is that most DevOps teams have little interest in truly understanding their application. There’s simply too much to do.

Ops By Any Other Name

The last couple of years have been interesting; I’ve talked to a couple of hundred folks, who largely have the title of DevOps, to understand what they do every day. There has certainly been a handful who perform this function as an integral part of the development team, and who have an intimate understanding of how their application and infrastructure works.

However, most appear to have simply renamed “IT Ops” to “DevOps.” That may be good for the resume these days, but it’s not great in terms of value delivered to the engineering team and the business overall. That also makes it not great for the future of the DevOps team more broadly.

The truth is that most DevOps teams have little interest in truly understanding their application. There’s simply too much to do. Between keeping their infrastructure running and providing tools to enable the engineering team to fix the application when it goes wrong, there’s not much time left. To make matters worse, organizations now following a ‘three pillars’ approach to observability are simply going to pile more plumbing and infrastructure work on this team. This will ensure that the supposed beneficiaries — Engineering & SRE teams — are, once again, disappointed. This may well be enough to call into question the value of the entire function.

Old Tools, New Data

The main problem that most encounter when adopting a modern approach to observability is outdated architecture. Most tools were simply not designed to ingest and reason about different types of event data. Nor were they designed for users to ask ad hoc investigative questions. They certainly didn’t expect users to query months, much less, a year’s worth of data.

DevOps teams spend much of their time managing and working around the not-so-elegant hacks that vendors have introduced to handle the diversity of data they now see from their customers — especially as they move to continuous delivery and microservices. Between trying to figure out how to parse YAML scripts on ingest, establishing elaborate tagging schemes, or managing storage costs by archiving ‘cold’ data to S3, there’s no shortage of time sucks. In today’s world, this is largely a waste of time and effort. It’s futile trying to get products to do things they were never designed to do. Worse, it represents zero value to users of the system — it is below the “value line”.

Democratizing Your Data

At Observe we took a different approach — by flipping the iceberg. We made the controversial decision to use a commercial database — Snowflake — as our underlying datastore. Yes, it costs us money, and by definition, it costs our customers money. However, it removes a massive operational burden from our users. They can parse data on the fly, they don’t need to worry about indexes, tags, the cardinality of data or god forbid, archiving data to lower storage costs.

The net result is that Observe requires less DevOps effort to get running, and to keep running. This also means that Observe’s engineering team can spend more of their time focused on providing a rich analytics environment for their users. Put another way, the engineering team spends more time above the ‘value line’ so that our users can deliver more value to their business.

It’s not just the traditional SRE and Engineering teams that will benefit. One of the key concepts that Observe has introduced is the notion of Resources — an abstraction layer above the machine-generated event data that makes it easier for users to reason about their applications and infrastructure. This opens up the data to a broader range of non-technical users, such as Customer Success teams and perhaps even Business users. The more people in the organization who have visibility into the data, the more valuable it is — and the more likely that data will be used to improve processes and organizational priorities like customer satisfaction.

The net effect is that observability can be tied directly to pain points the business needs to address and will increasingly become a critical part of business success in the future. What does this mean for DevOps? If you look on LinkedIn today there are approximately 81,000 DevOps job posts. As an industry we simply can’t hire our way out of this problem — there has to be a better way. Observe is that better way.