How to Formulate a New Experiment Idea
Learn the template we use at GrowthHackers
What is an experiment idea?
An experiment can be understood as actions taken in the hopes of validating a hypothesis. In a business context, it is a practical test that by taking action A, it will help reach goal B. Companies that are learning-driven are deploying experiments to solve business problems across every stage in their customer journey.
"[…] a company’s ability to create and refine its products, customer experiences, processes, and business models — in other words, to compete — is deeply affected by its ability to experiment. In fact, no innovation can exist without first being an idea that is subsequently shaped through experimentation. Today, an innovation project can involve hundreds or thousands of experiments, all with the same objective: to learn through rounds of disciplined testing, whether a business idea holds promise for addressing a customer need or problem." Experimentation Works book, by Stefan Thomke
While members of growth teams are more likely to be on the same page on how to formulate and propose new ideas when you democratize ideation and break silos, idea submissions from other departments will pop up in all sorts of formats. Navigating through a non-standardized idea backlog might make prioritizing and operationalizing experiments challenging.
New Experiment Idea Template
This article will help with an essential part of experimentation that allows your team to build an idea backlog full of insights and hypotheses ready to be prioritized and tested according to your business goals. To do so, I will dissect the template for a new experiment idea we use here at GrowthHackers.
“Ideas are the rocket fuel of growth. And you need a pipeline of them delivering a steady flow. Ideas should be submitted to an idea pipeline, following a templated format by which they should be submitted. It’s important to standardize the format so that ideas can be quickly evaluated, without the team needing to ask lots of questions.” Sean Ellis and Morgan Brow, Hacking Growth book
To do so, I will be using a fictitious meal delivery company to illustrate concepts. In this case study, our meal delivery company has checked their app usage data and separated their users into cohorts based on app usage, profile and purchase history and learned that the cohort with the most engaged and highest spending users take a few actions in common after they install the app, and have a similar profile.
They have defined their north start metric as the weekly number of orders. To support the north star metric, they have set a few objectives. To reach one of them, a member of the growth team had an idea.
North Star Metric: Number Of Orders
Objectives:
- Get existing customers to order more
- Increase the number of orders from new customers
1. Idea Name
Ok, so the first element of the new idea template is what the name suggests. The idea should be named as clearly and concisely as possible. We find that keeping an idea name brief makes discussing and locating an idea in the future much easier. For the purpose of this exercise, we will call this idea “Favourite Restaurants Notifications”.
2. Idea Description
Next, it is time to describe your idea in detail, with the what, the whos, the whens. It should contain how this experiment will be run, what is the sample for the experiment, what will be created, what resources are needed, and under what merits the idea should be taken forward. It is the rationale behind this idea: what proprietary data, benchmark, or customer interview justifies this test? What kind of test will be run?
"In determining whether an experiment is needed, managers must first figure out exactly what they want to learn and measure. Only then can they decide if testing is the best approach to achieve their answer and, if so, what the scope of the experiment should be.” Experimentation Works book, by Stefan Thomke
For our meal delivery app, the description should look like this: “We have identified that the user cohort who orders more often do so from the same restaurants they have ordered from previously, therefore, we should test notifying users when there is a promotion in a restaurant where they have placed a previous order and rated 4 stars or above. This test will be done over a period of 2 weeks with a control group from cohorts A and B and analyzed against the results from the control over the same period.”
3. Hypothesis
The experiment hypothesis is the expected cause and effect resulting from your experiment, what you are hoping to achieve if your test is successful. Quantifying the expected gain can help teams to have a clearer idea of success.
Calculating an expected lift can be done by using results from previous experiments, external benchmarks, or by an estimate based on the scale of your test. Including a quantitative prediction is preferable, but not mandatory, especially for teams who do not have a history of previous tests.
The more you test, the more accurate your hypothesis will become. Your hypothesis has to be phrased in a way that allows for it to be proved or disproved.
As an example, our made-up delivery app company could have as a hypothesis “If we can get other customers to behave like cohort A by reminding them of the restaurants they like, we can expect a lift in the number of orders”.
4. Objectives and Metrics
This is where you disclaim what metrics should be tracked to evaluate the outcome of your test. This metric should be connected to your objectives, which should, in turn, support the achievement of your North Star Metric. To learn more about how to set your goals, have a look at this article: The 3 Layered Cadence of KPIs of a Growth Strategy.
According to Sean Ellis, you should “identify metrics to be tracked by looking at the metrics “downstream” from the experiment that will be impacted”. In our case study, the metrics to be measured could be:
- Number of orders made from notifications from this campaign
- % of users who received a notification and completed an order right after
- Average number of orders from the experiment group
- Average number of orders from the control
These metrics will help the team assess if they have validated their hypothesis.
5. Prioritization
To help organize your experimentation pipeline, you can also suggest a prioritization grading according to your chosen prioritization framework, which can be adjusted during the growth meeting, when you are discussing the logistics and resources available to your team. Here at GrowthHackers, we use the ICE score prioritization framework, where we grade ideas for Impact, Confidence, and Ease.
6. Related Ideas
When suggesting an idea that relates to a past experiment, or to other ideas that are still waiting in your pipeline and relate to the same goal, it is a good idea to either tag ideas so they can be easily grouped, or link the other ideas.
By doing so, you will be able to filter related ideas that pertain to the same growth lever and objective and easily get a full picture of past tests and existing ideas for that topic when suggesting a new idea.
The Full Picture
Below you can see what this information looks like in our new idea template at Experiments by GrowthHackers. Once your team is briefed on your current goals and knows how to suggest ideas that can be understood by anyone in the team, you will soon have an idea pipeline to prioritize and test according to your objectives and resources.
How does your team suggest ideas in your company? Have you tried a similar process to make ideation easy?
Share it in the comments!
