Touching the unknown

When you work on an innovative new product there is lack of data. You will do research to validate your value assumptions. The truth is as long as this product doesn’t hit the market you can never be sure if and how it will be used in reality.

To overcome this problem, agile teams deliver software in increments. They follow patters such as the build – measure – learn cycle or talk about minimum (= next) viable (= best) products (= test) and include continuous customer feedback loops (Voice of Customer, Friends and Family feedback, etc.) to mitigate risk and build up levels of confidence.

Still, it happens that the real usage is not what was validated over and over. KPIs are not met, and people wonder where they took a wrong turn. In the worst case this is where the invention of narratives starts, and we identify hypothetical reasons (or scapegoats) why a certain result was not achieved in order to keep going. Don’t start spinning a yarn! Even if it saves you in inconvenient situations, it will lead to disaster. If not today, then tomorrow or the day after. It always fires back.

In the beginning you are more of an explorer who set out to discover unchartered territories. Therefore, you become a cartographer who maps out those new regions and transfers the unknown to the known.

In need of a navigator

Becoming a cartographer is not enough. We also need a navigator as we cannot trust our own gut feelings and assumptions. We need a navigator that is objective, always.

To go there, let us first talk about a navigational device with a certain superpower: a device that provides evidence, be it convenient or inconvenient news. Real data that is hard to deny or to fantasize about.

A device everyone has agreed upon to be able to steer us in troubled waters. Be it business stakeholders, product owners or agile teams. A device so powerful that it does not only have they buy-in of everyone once it is used but that is additionally a shapeshifter: it can be adjusted to almost any new situation.

Well, don’t stop reading. I am not making something up. This device truly exists. I am also quite convinced you already know about it the one or other way and the whole magic is only about putting attention to it.

By the way, an Astrolabe (Greek: ἀστρολάβος; literally: “star taker”) is an astronomic device that served as physical model of heavily bodies. For the ancient astronomer it was almost a Swiss Army Knife or the famous “eierlegende Wollmilchsau” in German colloquialism that refers to a mythical creature that can do everything. Just think of Chuck Norris, but way, way back. As a precursor to the Sextant, it allowed to determinate latitude and the explorer of worlds would utilize the mariner’s astrolobe in ancient times – an early spinoff product.

What are outcomes?

The navigational device I am talking about are outcome metrics. To make use of it, we must first understand the difference between output, outcome and impact.

Output

We have ideas (opportunities) that are transformed into concrete working software.

In an agile team, costs (human resources, infrastructure, etc.) and time (sprint duration) are fixed. Hence, the scope (deliverables) becomes the zone of influence.

Output is the very concrete working software. Output is great, because we can predict it and after each sprint, we can re-check the effort that was needed. We know everything about time and cost spent. There is nothing magical about it – it’s simple elementary school math.

Additionally, we know how we build the software. Therefore, the activities – the way how we created the working software – is well known to us. In an agile setup we have a scrum master role that helps the team to improve on the activities. Retrospectives are meetings where process optimizations, etc. are discussed and concrete actions are agreed.

Outcome and impact

While we know everything about output, initially we don’t know anything about outcome. Because outcome happens outside our organization (Unless we build working software for internal users but even then, the users are usually sitting in another division, department, etc.).

Additionally, there is a time difference: we estimate output in sprint refinements and adjust the estimation after each sprint. Therefore, a team knows precisely the effort spent. Compared to output it takes as a long – sometimes a very long – time to fully understand outcome.

When we think about idea bias – the simple fact that we overvalue our own ideas or the ideas of the social group (team, initiative, project…) we belong to creates a real problem: If we do not continuously validate the outcome, we might keep delivering working software that is not used as imagined or – worse – results in sunk costs.

The right navigational device

Well, Columbus failed. He never discovered the intended trade route to India going westwards. He was convinced until his death he had found a route to “Las Indias,” believing he had sailed down the coast of Asia. He could not have known that another ocean lay ahead to cross. If the organization or entrepreneur is capable of truly understanding the full impact of one’s finding is another topic worth thinking about.

We remain on the level of the navigational device. Our target is to design and build the right digital product or service. Our goal is to stay on course to make those discoveries in the first place.

As the ship’s boy turned the sandglass every half-hour to keep track of time, the nocturnal was used to determine midnight and the quadrant supported orientation, we will utilize outcome metrics to be able to navigate towards our goal.

A word on metrics

When we think about agile software development we deal in essence with three main types of metrics:

1. Output metrics
2. Impact metrics
3. Outcome metrics

All those metrics matter but each has its own qualities. We must be aware of the limitations and power of each metric to make use of them.

Output metrics

Output metrics are important to measure the performance of our activities.

• For example, to measure delivery speed or if stories were implemented in an efficient order we use story breakdown charts,
• or we improve the work process with team retros and agreed action points.

Those metrics will not tell us anything about the usage of the working software. Just building software efficiently does not tell us if and how the software is used by our users.

Impact metrics

Impact is about understanding what will drive business results.

Think about: increasing the revenue, decreasing costs, increasing market share / customer adoption, increasing revenue from existing customers, increasing shareholder revenue, increasing service delivery / productivity, strengthening the brand, increasing life-time value, decrease cost of acquisition (marketing, salespeople), increase monthly return in revenue (MRR), etc.

Examples for impact metrics:

• Cost / Benefit ratio: compare the investment (efforts spent) to the expected value. To give a high-level example, building a certain feature might cost 80 person days and 5 person days per year for maintenance. Then you can compare those efforts to the expected return.
• Business Impact potential: additionally, you can calculate the potential return and how likely is it to be in the lower / middle / high ranges?

An impact metric is a generic way to measure business success and provide us with comparable – although high level – quantifiable data (increase / decrease).

Most of the time impact metrics as success metrics are too abstract. They cannot easily be translated into concrete features.

Think about increase revenue: how should this be translated to something particular?

Therefore, impact metrics can serve as a basis to define specific outcome metrics that can then be translated to output (working software). Impact metrics can act as guiding stars but to travel, we must first understand how to read them and what concrete actions we must take to follow their path.

It is all about making things concrete and applicable.

Outcome metrics

Outcome metrics measure concrete changes in user behavior.

Below a simple example from Josuah Seiden (cf. Seiden 2019:32):

Measuring the number of people calling tech support is benchmarking your output against concrete user behavior. You know exactly what effect your output had on customers. Additionally, you can translate those outcomes to impact (business metrics). This is the full loop.

Another example:

Again, measuring the engagement of students towards the existing and new course offerings is benchmarking your output against concrete user behavior. You know exactly what effect your output had on the students and you can translate those insights to impact – in this case improved educational standards.

Be aware that not everything that brings value to users brings also value to your organization. We could also get blindfolded by looking at outcomes alone.

Make metrics concrete

It is important to understand right from the beginning – when a (business) opportunity is identified – what benchmark can be used to measure success or failure along the whole journey, from the initial idea to the mature product.

There is no specific rule what metric framework to use. Personally, I prefer OKRs as their structure to first think about objectives and subsequent key results serves our purpose well. What matters is that the key results are behaviors / actions on your user side that you can monitor on a regular basis.

1. Set clear Objectives: they should be ambitious, inspiring, and aligned with your organization’s goals. Objectives should be qualitative and provide a clear direction.
2. Define Key Results: they are specific, time-bound, and measurable. Key results indicate progress towards the objective. Typically, two to five key results are set for each objective.

It’s a collective endeavor

New territories lurk everywhere: It took 48 years until the particle described by British physicist Peter Higgs was confirmed at the CERN Large Hadron Collider. There, more than 3000 physicists collaborate nonstop around the world to make such discoveries possible.

A whole village of physicists was needed to validate or invalidate this particle. As a captain needs his crew or innovative ideas feed on centuries of thought, we are standing on the shoulders of our predecessors. Discoveries never happen out of the blue.

The benefits of outcome metrics

Many navigators

The metrics itself are the navigational device. Additionally, we need navigators who take decisions based on these metrics.

In large companies a lot of different roles work together. There will be business sponsors, product managers, delivery managers, stakeholders of many different flavors, agile teams or even teams of teams, etc.

Each of these roles will have to face the following risks in one or the other way:

• Value risk: will users use / buy this product?
• Usability risk: will users be able to use this product?
• Feasibility risk: will we be able to build this product with our given resources (people, time, skills, technology)?
• Business viability risk: will we have a return of investment?
• Product specific risks such as regulatory risks or ethical risks and so forth…

Many navigators but only one navigation device

Outcome metrics bring a lot of power to the organization and each individual:

Using outcome metrics will create a shared benchmark across the organizational structure to focus on the same end-goals. Understanding what user behaviors drive what business impact will create a customer centered focus on all involved levels.

• Alignment: They align the efforts of different departments and teams on the same end-goals.
• Clarity: They provide clarity on what defines success or failure to make the shared objectives tangible.
• Decision-making: they inform decision-making by highlighting success and failure
• Accountability: The hold individuals and teams accountable for their contribution to shared goals.

Shifting from abstract KPIs or process-oriented metrics alone to outcome-based measurements shifts the focus how we collaborate and work. A shared benchmark across organizational structures fosters cross-team collaboration and streamlines the whole product development process.

• Greater Flexibility: Teams can adjust and tailor their methods to best achieve the desired outcomes, fostering a flexible work environment.
• Cultural / Mindset Shift: There is a shift towards innovation / experimentation and shared ownership rather than fulfilling given requirements.
• Efficiency: Focusing on outcomes can streamline operations and create processes for gaining and including continuous feedback loops (customers and process) on all levels of the organization.

Empowerment happens on each individual role. If there is clarity on what output creates the desired outcomes it fosters transparency. Each individual and team can immediately see their contribution towards the objective. It fosters a shared responsibility. Outcome metrics shift the focus from merely following prescribed tasks to achieving defined results, which naturally distributes power and control.

• Empowerment: By focusing on outcome, employees at all levels are empowered to make decisions that contribute to the strategic goals, rather than just following processes or requirements.
• Communication: They facilitate better communication about goals and strategies.
• Responsibility: They assign responsibility for specific outcomes to teams or individuals, regardless of their position in the hierarchy, fostering a sense of ownership.
• Innovation: With a clear understanding of desired outcomes, individuals at all levels are encouraged to innovate and experiment to achieve those outcomes.

What’s next?

Use this slide deck to share the power of outcome metrics with your peers:

Continue reading more about product thinking: