Anchor to the Invariants
Hampered Innovation--A Most Common Pitfall
Many of you reading this will have lived some version of the following story, some of you many times: You're on a product or service team and word comes down that it's time to prioritize adopting the new shiny thing. Could be anything--some new experimentation framework for testing surface changes, building a new high-tech feature (LLM furniture shopping in rural PA, anyone?), whatever.
The team gets fired up, implementation, integration, delivery. The glorious future is here. But after the sugar rush from the celebratory sheet-cake wears off, your metrics are slipping. Or have changed in an unexpected way. Maybe complaints are up, or even adoption and customer satisfaction are. Good, bad, ugly--it doesn't much matter, the team can't explain the delta. There are many flavors of this, some include:
-
Customers seem happier but aren't using the new feature the way we intended.
-
Customers are upset despite everything working precisely as we expected it to.
-
Our new experimental data is accurate and yet changes made in response are driving metrics in the wrong direction.
-
A novel system or security incident strikes, while it's obvious something bad is happening, tracking down what proves difficult.
The metapoint is that one of the most common impediments to innovation is making changes--even very good ones--without having a sufficiently mature understanding of the space to evaluate their impact.
In this essay, we're diving into how knowing yourself--customers, company, team, domain, system, product, market, etc.--with rigor and discipline both protects you from some harms and empowers advancing the plot for your business in the context of unknown unknowns.
This is non-negotiable table stakes for farming anything of value out of Seeing in a Strange Wilderness.
Brief Remark on Tension + Acknowledging Objections
Some readers are bristling at this point, suspicious that rigorous, disciplined self-knowledge detracts from forward progress in a game where you either evolve or die. Or perhaps that by canonizing a powerful impulse for operational excellence toward such self-knowledge, it can become a club used by entrenched middle-managers to protect kingdoms and stifle innovation even further.
If you're in either of those camps--you're right. What I advocate for here can be very dangerous if left unchecked by an implementing organization. Ensuring that the right balance is struck on tensions like this is a core function of leadership--a topic I look forward to treating specifically in a later essay.
For now, let it suffice that cultural tradeoffs are much heavier than system tradeoffs--requiring judgment calls fueled by conviction, courage, and tenacity in leaders. The meta-metapoint (we'll halt the regression here for now!) is that there is no system or magic algorithm for effectively driving organizations. Everything is pathological at the wrong dose and all ingredients react with each other, sometimes in catastrophic ways.
</tangent>
Operational Excellence as Reality Anchor
The most common take on operations excellence that I encounter is some flavor of "we've got some metrics that tell us how we did, and some of them have alarms to trigger a response if something goes wrong." Sure, can't argue very well against the utility of that. Where this falls short is in that that is the whole story--historical data and some reaction triggers.
This is an impoverished stance, ops can give you so much more.
Company Meets World Where Future Becomes Present
Before we can fruitfully dive into a richer view of operations, it's necessary to sketch out exactly what I mean by the "self" here. There's a seductive impulse toward forgetting that even a company facing annihilation due to drastic change--think a firm that doubled down on mass-market horse products in the face of the Model T--has invariants.
This especially bears explicit treatment today--many folks are overwhelmed by a sense of rapid changes in the world. Remember that despite all of the excitement, there is much that isn't changing--at least not yet. This is the foundation of evolving to meet the present.
Some variation of the following is relevant in your domain:
-
Company (or organization). History, culture, policies, brand, narrative, placement, etc.--the stuff that comprises the body without flesh giving shape to everything else.
-
Customers. Buyers, users, recipients, clients, partners, etc.--whoever it is transferring money or some other resource into your world in exchange for some value that you provide.
-
Product/Service Surface. The details of what you actually offer to your customers.
-
Systems. Machines, information resources, factories, or other structures by which the company operates products/services to provide value to customers.
-
Processes. Interfaces between your people and your systems taken holistically to be the meat of what actually happens during the course of operating in your space.
-
People. Your employees, leadership, vendors, counsel, contractors, etc.--the human beings who use your processes to operate your systems and so forth.
Categorically speaking each of these exists in some fashion for what you're doing until you cease doing it. This is the invariant part--even where all employees quit at once, people remain a first-class concern. All firms have the opportunity to deeply understand, if not actually control, each of these aspects of their operations.
Contrast the above with:
-
Your legal/regulatory landscape,
-
Introduction of disruptive technologies,
-
Social/political upheaval rewriting the fabric upon which you do business ,
-
Black swan events,
And similar--in opposition to "self" let's call these "other".
The "other", the waves in the art above, ultimately demands reaction by its character. The "self", however, can engender proactive action through deep understanding; not all ships pilot the same way. Notice that the ship doesn't make sense without the waves--solipsism is not the answer!
(Neither is nihilism, but we'll deal with that in a later part of this series.)
Now that we have a fuzzy line between "self" and "other", let's get to the point: using mastery of the "self" to detect and proactively respond to aspects of the "other" not easily understood by direct analysis.
Know the "Self": Rigorous, Mature, Improving
So practically speaking, what does a high-degree of "self"-knowledge look like?
Let's start by sketching something like an ideal state--and acknowledging that we're not going to achieve it, which is fine. Operations demand continuous improvement, and face the full spectrum of social and physical idiosyncrasies of reality. This isn't a solvable problem--it's all about achieving a defensible baseline from which you can work, improve, and evolve without going under.
I use examples derived from my experience doing tech ops on large Internet platforms, however the analogues to other domains are rich indeed.
Abstract Map. Have developed, tested, and established mechanisms for adjusting as comprehensive a map of the levels of abstraction present in your domain as possible. For a product team, that list might look something like:
-
Network
-
Infrastructure
-
Data Flow
-
Application
-
User Journeys
-
Corporate Outcomes (e.g. Money Flow)
-
Cross-Product Coherence
First-Order Metrics. For every level of abstraction relevant, capture metrics directly at their source. "First-order" here means not derived from other values. A few examples, continuing the above:
-
Latency
-
CPU/memory load for the nodes
-
Read/write error rate
-
Uptime heartbeat
-
Login success rate
-
Naive revenue from successful checkouts
-
Continue rate for users entering funnel from a sister product surface
Nth-Order Metrics. Layers of derivative metrics building upon first-order data. These are your statistics, calculations, compositions, and so forth--probably comprise the majority of your information but are wholly dependent on the first-order data to approximate anything like truth. Instead of specific examples, here's my general take on mature Nth-Order Metrics:
Being able to take nearly any change in system or flow state and draw a bright, clean, easy-to-understand line from that change to other changes in the overall picture. For instance, how much revenue is lost on your e-commerce platform when latency increases by 1.5%? Answering that in the ideal state is not a research project but a button click. Where that isn't possible is the exception, not the rule--and is therefore a trigger for human investigation.
Additionally, the baselines for every metric at every level of abstraction are well-understood--as are the error statements for all of them. Operators, analysts, and leaders across the space can easily determine whether a fluctuation in one metric is explainable by changes in others or not. You end up either with a closed epistemic loop or a heuristic pointing to (likely) fruitful areas to begin an investigation.
Information & Intuition. The matrix of metrics in our ideal state then acts to minimize the attention paid to matters of little consequence and the time it takes for attention to fall on interesting or grave events. The team(s) spend time during every ops cycle disconfirming their trust in such minimization and adjusting accordingly.
We end up with a highly effective quantification at every level of abstracted that pushes human attention where it is most useful--all the while not falling into complacency by ensuring that the matrix is sanity checked and tested by people on a regular basis for both quantitative and qualitative reasons.
Finally, couch all of the above in something like the following:
-
Interrupted telemetry, beyond some very low threshold, wakes someone up 24/7 365. Fluctuations and perturbations below the threshold are investigated by on call staff during lulls in the action.
-
Metrics reviews, operational readiness reviews (Amazon-style, details in another post sometime), and post-incident analysis are canonized as first-class cultural rituals AND as part of the blessed work celebrated in both performance reviews and promotion packets.
-
Cross-functional leadership works to maintain a common language for interpreting metrics, intervenes to restore the common view when skew arises, and takes very seriously any intuition that something is wrong.
Lovely state of affairs to aspire to! Enough idealism, let's get back to the surly bonds of Earth.
The Dweller
Regardless of maturity or the particular warts of your specific context (we all have them), the key moment in the ideal state above is when you can't explain, with what you have, a change that you're seeing. We have to be careful with "explain" here--I specifically mean robustly explain, with quantitative and qualitative evidence, the connection between a change and other world-state that is sufficient to act on at that time.
Many organizations have data scientists, data analysts, operations analysts, and similar deployed in various ways to support farming information and scaffolding decisions. I've developed a specific flavor of such a role that I find to be tremendously useful in epistemically-fraught, high-complexity, difficult spaces--such as trust & safety, security, or marketing.
I (half-jokingly) call the role concept the Dweller, after the highly-skilled employees in cyberpunk fiction who focused on defense of a location in cyberspace. The Dweller does not belong to a centralized functional organization and is directly embedded within their domain of responsibility.
This person has quantitative depth (think statistical physicist, mathematician, or actuary), develops an fluency in their quantified domain, and serves as the subject matter expert for their team on metrics, baselines, and how it feels as the numbers change.
When not investigating a gap in the matrix, the Dweller drives the group's metrics review, works with leadership and operators to provide complex inputs as necessary, and oversees maintenance on dashboards, alarms, and other facets of the quantified domain.
I find that having a dedicated person, a sort of data-clergy to the data-lay staff if you will, to make the quantified domain their entire focus is tremendously useful. On call engineers and analysts have other responsibilities and incentives. The Dweller(s) maintain constant focus on baseline, changes, telemetry rot, examining little things for patterns over time, and generally developing a feel for your data and delta.
Such focus and depth uncovers threat and opportunity alike.
If you can afford it, having such a person or team greatly enhances your group's potential for finding something novel in the "other" by sustaining mastery over the "self". If you can't, finding ways to engage your team in some of the Dweller activities, even on a start/stop or cyclical basis, can farm some of the benefits.
Views of "Self" and "Other" In Symbiosis
None of this is to say that those in your company studying the "other" aren't an important piece of lifting the fog of war. My thesis here is that deep understanding of "self" can help shed light on aspects of the "other" that are resistant to analysis. Information about "self" and "other" flowing into each other--challenging, correcting, unstucking, inspiring, warning--starts approaching something like a realized, rich view of operations excellence.
Whether we understand the cause or not, change happens at every level of abstraction constantly. By going deep on the stuff that you can see particularly clearly, you may just notice tell-tales of change stemming from sources you may not yet realize exist. You are somewhere on this gradient already, I encourage you to find ways to push farther in your context.
Finally, Ready for Shiny Things
The degree to which you can achieve substantive and stable maturity along these lines is roughly the degree to which you can practically benefit from the tools, approach, and conceptual framework that I will detail in Seeing in a Strange Wilderness. There is no substitute for knowing your own house, even in fantastical worlds haunted by machine consciousness or in some Gattaca-esque future.
Merely knowing, however, is not enough--you must act.