If you’re supporting a technical product that has achieved some level of market fit and success, you are likely running “experiments” (like A/B tests) to ensure that the product iterates in the right direction.

Unsurprisingly, when you foster a culture of experimentation your organization can trust that it is making the best decisions to support its goals, and healthy experimentation is the key.

What makes experimentation “healthy” or “unhealthy”?

Many factors can affect experimentation within your organization, and the importance of these factors might differ from organization to organization based on how that organization views experimentation.

At a high level, healthy experimentation is:

  • Trusted. Stakeholders trust the data from experiments that your organization runs. It is clear that the data collected is exactly what we intended to target, and we can prove that.
  • Stable. Technical issues are rare. We’re able to easily run experiments without delays or the need to disable and restart due to unforeseen issues related to configuring our experiments.
  • Observable. We’re able to tell at a glance that our experiments are working as we expect. Users are being exposed to treatments, exposures are balanced, crossover users are minimal, etc.

This, of course, means that unhealthy experimentation might be:

  • Untrustworthy. There is a history of issues related to running experiments. Our stakeholders don’t always trust results and must do additional analysis to confirm that the results of an experiment represent reality.
  • Fragile, unstable, and/or high maintenance. We often have issues defining and running experiments. This leads to delays and extra work resolving these issues, and heavily influences the level of distrust in our experiments.
  • Opaque. We don’t know if our experiments are working as intended, or worse, we don’t even know how to check. We usually don’t find out about these issues until a stakeholder highlights some issue with the data.

Unhealthy experimentation can cause lasting damage

Poor experimentation health has an incredibly harmful side effect: distrust.

When an organization loses trust in it’s experimentation process, it might start to question its decision making (past and future). How can we be sure that we’ve historically made the right decisions for the business and our customers if we can’t even be sure that our experiment results are trustworthy?

Healthy experimentation drives healthy innovation

When we can use data to drive our decision making, we can prove that the decisions being made are the right ones for the business. For this to be effective, however, we need to trust that data and be able to understand how it fits in the broader picture.

This in turn can help inform where our next and most impactful investments may be.

Healthy experimentation is an accelerator

When experiments are in a healthy state, we move faster. When our experiments “just work”, when we can observe not just their health but also how the data tracks against our hypothesis, and when we trust the process from end to end, we can iterate more quickly and intentionally.

Every experiment that we run that doesn’t require extra debugging or data manipulation to identify an outcome means that we’re able to instead focus that effort on our next experiment. Healthy and trustworthy experimentation means that each experiment is an iteration on our product that teaches us something new and helps inform our next set of hypotheses.

Like compound interest, the long term benefits of compounding these learnings and unlocking the next set is an accelerator for the business, and can even be what gives your organization the competitive edge.

The big picture

Individual experiments in isolation can be incredibly informative, but the archive of learnings across experiments is invaluable.

When experiments are trusted, stable, and observable (both while running and historically), we grow this set of learnings more quickly and can apply it to new explorations more easily. This can inform the “types” of efforts that we pursue or the business level metrics that we should target for highest impact.

Final thoughts

Experimentation is critical to an evolving product. That said, unhealthy experimentation can be actively harmful to your organization and the business. This harm can come in the form of distrust in your organization’s experiments, instability in your experimentation framework, and difficulty understanding why all of the above might be happening.

As a result, unhealthy experiments can make the process of decision making feel like wading through mud. When we finally do come to some conclusion, its usually a bit unsatisfying due to the cloud of uncertainty hanging overhead. Worse yet, if that conclusion turns out to be incorrect, our time is spent undoing the damage rather than moving on to the next initiative.

Healthy experimentation and a culture that puts value on experimentation are accelerators, allowing your organization to learn more about its customers and product and to lean into these learnings to ensure that it is making the right decisions. Experiments that run more quickly and without issue allow us to make decisions more quickly, which in turn allows us to start the next iteration and experiment.

This acceleration is similar to the notion of compound interest, allowing us to leverage and build on past success and learnings to achieve even greater returns.

Next up

This article is the start of a series on experimentation within a technical product. The next chapter will explore trust as a fundamental pillar for experimentation, and how you might achieve and maintain it.