Move fast, break things and take chances to make experiences truly remarkable.

In this post, I’d like to explore the value of experimentation and share a few ideas to help you get into a position where you can run experiments in your product teams. 

When you work in a scaling organization, you slowly notice that the fear of failing grows too. A mistake no longer affects 300 users; it affects 10,000. Much like the curved hockey stick growth chart, the price of your decisions also balloons. 

It’s easy to over-exert caution because safe is good, right? Wrong. Playing it safe keeps you on a steady path of ticking boxes, dotting i’s, and crossing t’s. Unless you’re building products for medical teams or NASA, you need to learn the value of experimentation. 

Table of contents

The benefits of experimentation at scale

An experiment can be as small as a button click or as big as a series of workflows in your product. And to address one of the biggest fears of experimentation as you scale (failure), here are a few reasons why running experiments is great:

  • Keep an eye on that bottom line. Experiments are far cheaper to run than long discovery sessions and provide quantitative data for you to share with leadership teams before fully investing in an initiative. 
  • Larger volumes of quantitative and qualitative data. You now have a large user base to gather more information from, faster. I can’t count the times in the past when I’ve sent out a survey or added a new interaction and got dismally small results. 
  • Fail fast experiments are one of the quickest ways to test. To quote the 3% rule, the products we build are defined by the 97% we don’t. The faster you can figure out what that 97% is, the better your competitive edge. 

Tips, tricks, and lessons from the trenches 

Get the lay of the land 

Small organizations with new tech stacks can move much faster than larger organizations with ancient sections of code that “magically” work but everyone is afraid to touch.  

So my first tip is: speak to the engineers! Understand what your cash cows and stumbling blocks are. Here’s what you want to find out: 

Expansion opportunities 

Here, we look at growth opportunities or additional investments that help with existing clients or expanding to new markets.

  • Learn which parts of the code base are doing well. Which areas are they really excited about and why? 
  • Is that area of the product something the company is investing in, and why? How does it align with the direction of the company? OKRs and product/company strategy documents are very helpful with this.

Risks 

But all the same, which areas of the code base make them shudder?  

What does it cost your product team to “keep the lights on”? Meaning how much time goes into maintenance versus building? 

  • Growth: What risks does it pose to the current growth of your product?  
  • Opportunity: What are the overhead costs? If you tried to run an experiment in that product area tomorrow, how long would it take to get it live? This will give you an idea of how much more you have to spend in development time before you can test your ideas. 

If your engineers could wave a magic wand, how would they account for these risks? Start a conversation with your engineering team on how you can gradually work towards mitigating the major risks and turning them into growth opportunities.

Outcomes and hypotheses

We learn so much during discovery and experimentation that it can very easily lead to going down the rabbit hole, ultimately increasing your time to value. 

Setting clear outcomes and defining the hypotheses you’d like to prove or disprove creates a clear direction for your end goal. Add metrics to that, and you have guiding points along the way that help you stay accountable and measure your progress towards building functionality that brings value to your users.

  • Outcomes: What change in user behavior are you trying to drive? Set the direction you want to go if all goes well with your experiment.
  • Hypotheses: What are you trying to prove or disprove by running your experiment? Ask yourself: How do I know that adding this button or new workflow will benefit my user? Make sure there is a clear line for you to decide if you’d like to persevere with investing in development or if you need to go back to the drawing board. 
  • Begin with metrics in mind. An experiment means very little if you can’t measure the impact. Before you run the experiment, make sure you have some form of reporting in place to measure adoption (e.g., clicks /interactions), time saved (does your user hit their objective faster?) and user sentiment (a CES survey or your preferred gauged scale of 1-5 or 1-10) that helps gather some indication that your users would really like the idea of you doing X.

A safe place to fail 

I work with an engineering lead that has a knack for taking people from a junior level all the way to leading their own team time and time again. I just had to ask him how he does it, and his answer took me by surprise: create a safe place to fail. People who are afraid to take risks and think outside the box will never try new things, never experiment and surprise you with amazing results.  

Innovation comes with risk. I hope you remember the element of risk that got you started because that’s the key to keeping your competitive edge. 
Aldair Borges profile picture

Author

Aldair Borges

Aldair joined Mews in 2017 as an integration specialist and has since moved on to being one of our most loved and talented Product Managers, focussing on improving the reservation process.