PUBLISHED
11 June, 2023
Growth Manager
Product experimentation is a powerful tool with a surprisingly low barrier to entry.
Don't believe us? Consider the fact that all you need to do to uncover 85% of UX issues is run a usability test with as little as 5 users.
In this guide, we'll dive into the actual processes, elements, and tools (like UXCam) needed for effective product experimentation. By the end, you'll be able to create hypotheses, run experiments, make targeted improvements based on the results, and more!
Let's get started.
Product experimentation (just like scientific experimentation) is the systematic process of testing ideas or hypotheses to measure their impacts—in this case, on customer satisfaction and product performance. It’s an iterative approach that allows you to try out different products, features, and designs to learn what works best for your customers.
Usability testing is one of the more common types of experiments.
Your goal with this kind of testing is to examine how easily and intuitively your product can be used by customers. It usually involves tracking usage and frustration signals, running 1:1 usability interviews, and conducting surveys.
As the name suggests, functional testing involves testing the functionality of a product. This could involve testing out different algorithms, features, or designs to determine which performs best in real-world scenarios.
Security testing ensures that your product is secure from malicious attacks and data breaches. This involves testing the authentication systems, protocols, or other security measures to ensure that they are up to standard and can withstand potential hacking attempts.
Say it with us—“correlation does not equal causation”.
Product experimentation helps to establish cause and effect rather than relying on correlation. By testing hypotheses in a controlled, methodical way, you can be sure that changes to your product are actually making an impact rather than being a side effect of other factors.
Product experimentation can also help you identify friction points in your product. By testing out different designs, onboarding flows, and other features, you can quickly determine which areas of your product are causing user confusion or frustration.
Behind every user action is a “why”.
Uncovering the motivations behind user behavior gives you a better understanding of your customers, which in turn can help you make more informed product decisions.
Experimentation allows you to test different hypotheses and uncover that “why”—no intuition or guesswork required.
Want to conduct an organized, productive product experiment? You need a clear goal.
Start by defining your desired outcome. What’s the purpose of this experiment? Are you trying to increase engagement with a feature? Reduce the time it takes for users to complete a task?
There are plenty of worthwhile goals to choose from—just make sure that you pick one that is achievable within a reasonable timeframe. Product experimentation should be practical rather than aspirational.
As I'm sure you remember from high-school science classes, you're also going to need a hypothesis—or a statement that will (hopefully) be validated by the experiment.
Your hypothesis should be relevant to your goal and must be specific, testable, and measurable.
For example, if you want to increase your in-app conversion rate, your hypothesis might be something like “Reducing UI freezes by optimizing app performance will increase in-app conversion rate by 5%”.
Coming up with a product experimentation hypothesis is not a matter of luck—it’s a systematic process based on data. Usually, an analytics tool like UXCam is the best way to get the insights you need to form worthwhile hypotheses.
Here's a quick overview of the process using UXCam:
1. Map user flows & funnels. Start by mapping out user flows and funnels—where are users doing what? Which segments of your app are most popular? Which user journeys lead to the best outcomes?
2. Look for bottlenecks. Bottlenecks are obstacles that prevent users from accomplishing their goals (and, subsequently, yours). Which areas of your app seem to be causing the most delays? Look for high drop-off rates.
3. Look for patterns in behavior. Zoom in on user behavior through session replays, heatmaps, and event logs to identify patterns associated with bottlenecks. UXCam's easy segmentation makes this process a whole lot easier!
4. Transform patterns into hypotheses. Finally, take the patterns you’ve identified and turn them into hypotheses that can be tested. Seeing lots of UI freezes just before dropped sessions on a certain screen? That might be impacting conversions, engagement, retention… the list goes on.
KPIs and metrics are crucial for product experimentation because they give you a clear way to measure the impact of your changes. Before you start experimenting, decide which KPIs and metrics you’ll be using to track success.
Generally speaking, there are four broad categories of metrics you should keep an eye on during product experimentation:
Acquisition: Metrics that track how users find and interact with your product (e.g., downloads, signups, website visits).
Engagement: Metrics that measure how users interact with your product (e.g., session length, time spent in-app, usage frequency).
Retention: Metrics that measure how long users stick around (e.g., churn rate, number of active users).
Conversion: Metrics that track the success of user journeys (e.g., task completion rate, conversion rate).
In addition to matching KPIs and metrics with your goal, you'll also probably want a mix of leading and lagging metrics. Leading metrics give you an indication of how your changes are currently impacting user behavior, while lagging metrics tell you how successful they will be in the long run.
Unique users that fail to sign up, unique users who experience a UI freeze, previous screens from Cart Activity, top next screens from Cart Activity.
Now that you have a goal, hypothesis, and KPIs all setup, it’s time to start experimenting!
Make sure you run experiments with a control group—this ensures that any changes you make are actually responsible for the shifts in user behavior.
As you go through the experimentation process, it’s important to pay close attention to the data. Monitor your chosen KPIs and metrics to see if they’re trending up or down, and adjust your experiment accordingly.
Not all data is good data. To make sure the data you’re collecting is significant and valid, here are a few things to consider:
Sample Size: You need to use a sample size that reflects your user base. If you're not familiar with confidence intervals and margins of error, use a sample size calculator to determine appropriate sample sizes.
Timeframe: Run experiments for long enough to gather meaningful data but not so long that the experiment becomes outdated.
Statistical Significance: Statistical significance is a fancy way of saying that your data is reliable—it's not just random noise. Again, if you're not familiar with statistics, you can use an online calculator to check for statistical significance.
Product experimentation is only effective when you have an abundance of high-quality data to work with. You need it to form sound hypotheses and to test those hypotheses.
UXCam makes it easy to collect and analyze data for product experimentation. Session replays let you zoom in on user behavior, while segmentation tools help you quickly filter data. That way, you can focus on the information that matters most—for example, patterns in behavior associated with user success or failure—and make more informed decisions.
Ready to do more with your product experimentation? Sign up for a free trial of UXCam today.
AUTHOR
Growth Manager
Ardent technophile exploring the world of mobile app product management at UXCam.
Stay up-to-date with UXCam's latest features, insights, and industry news for an exceptional user experience.
How is the increase in mobile use changing the landscape of the product management role, and what are the differences between traditional product management and mobile product...
Explore examples of Product Management KPI Dashboards and learn how to effectively use them to drive business...
You can attend the best product & UX conferences online this...