A Practical Guide To Product Experimentation

What is product experimentation?

Product experimentation (just like scientific experimentation) is the systematic process of testing ideas or hypotheses to measure their impacts—in this case, on customer satisfaction and product performance. It’s an iterative approach that allows you to try out different products, features, and designs to learn what works best for your customers.

Different types of product experiments

Usability testing

Usability testing is one of the more common types of experiments.

Your goal with this kind of testing is to examine how easily and intuitively your product can be used by customers. It usually involves tracking usage and frustration signals, running 1:1 usability interviews, and conducting surveys.

Functional testing

As the name suggests, functional testing involves testing the functionality of a product. This could involve testing out different algorithms, features, or designs to determine which performs best in real-world scenarios.

Security testing

Security testing ensures that your product is secure from malicious attacks and data breaches. This involves testing the authentication systems, protocols, or other security measures to ensure that they are up to standard and can withstand potential hacking attempts.

Why is product experimentation important?

Causation over correlation

Say it with us—“correlation does not equal causation”.

Product experimentation helps to establish cause and effect rather than relying on correlation. By testing hypotheses in a controlled, methodical way, you can be sure that changes to your product are actually making an impact rather than being a side effect of other factors.

Spot friction points

Product experimentation can also help you identify friction points in your product. By testing out different designs, onboarding flows, and other features, you can quickly determine which areas of your product are causing user confusion or frustration.

Improve customer knowledge

Behind every user action is a “why”.

Uncovering the motivations behind user behavior gives you a better understanding of your customers, which in turn can help you make more informed product decisions. 

Experimentation allows you to test different hypotheses and uncover that “why”—no intuition or guesswork required.

How to do effective product experimentation

Step 1: Set realistic experimentation goals

Want to conduct an organized, productive product experiment? You need a clear goal.

Start by defining your desired outcome. What’s the purpose of this experiment? Are you trying to increase engagement with a feature? Reduce the time it takes for users to complete a task?

There are plenty of worthwhile goals to choose from—just make sure that you pick one that is achievable within a reasonable timeframe. Product experimentation should be practical rather than aspirational.

Step 2: Build a hypothesis relevant to your goal

As I'm sure you remember from high-school science classes, you're also going to need a hypothesis—or a statement that will (hopefully) be validated by the experiment.

Your hypothesis should be relevant to your goal and must be specific, testable, and measurable.

For example, if you want to increase your in-app conversion rate, your hypothesis might be something like “Reducing UI freezes by optimizing app performance will increase in-app conversion rate by 5%”.

How to come up with a hypothesis

Coming up with a product experimentation hypothesis is not a matter of luck—it’s a systematic process based on data. Usually, an analytics tool like UXCam is the best way to get the insights you need to form worthwhile hypotheses.

Here's a quick overview of the process using UXCam:

1. Map user flows & funnels. Start by mapping out user flows and funnels—where are users doing what? Which segments of your app are most popular? Which user journeys lead to the best outcomes?

2. Look for bottlenecks. Bottlenecks are obstacles that prevent users from accomplishing their goals (and, subsequently, yours). Which areas of your app seem to be causing the most delays? Look for high drop-off rates.

3. Look for patterns in behavior. Zoom in on user behavior through session replays, heatmaps, and event logs to identify patterns associated with bottlenecks. UXCam's easy segmentation makes this process a whole lot easier!

4. Transform patterns into hypotheses. Finally, take the patterns you’ve identified and turn them into hypotheses that can be tested. Seeing lots of UI freezes just before dropped sessions on a certain screen? That might be impacting conversions, engagement, retention… the list goes on.

Step 3: Use KPIs for measurement

KPIs and metrics are crucial for product experimentation because they give you a clear way to measure the impact of your changes. Before you start experimenting, decide which KPIs and metrics you’ll be using to track success.

Generally speaking, there are four broad categories of metrics you should keep an eye on during product experimentation:

• Acquisition: Metrics that track how users find and interact with your product (e.g., downloads, signups, website visits).

• Engagement: Metrics that measure how users interact with your product (e.g., session length, time spent in-app, usage frequency).

• Retention: Metrics that measure how long users stick around (e.g., churn rate, number of active users).

• Conversion: Metrics that track the success of user journeys (e.g., task completion rate, conversion rate).

In addition to matching KPIs and metrics with your goal, you'll also probably want a mix of leading and lagging metrics. Leading metrics give you an indication of how your changes are currently impacting user behavior, while lagging metrics tell you how successful they will be in the long run.

Unique users that fail to sign up, unique users who experience a UI freeze, previous screens from Cart Activity, top next screens from Cart Activity.

Step 4: Conduct your experiment and monitor the data

Now that you have a goal, hypothesis, and KPIs all setup, it’s time to start experimenting!

Make sure you run experiments with a control group—this ensures that any changes you make are actually responsible for the shifts in user behavior.

As you go through the experimentation process, it’s important to pay close attention to the data. Monitor your chosen KPIs and metrics to see if they’re trending up or down, and adjust your experiment accordingly.

How to make your data significant and valid

Not all data is good data. To make sure the data you’re collecting is significant and valid, here are a few things to consider:

• Sample Size: You need to use a sample size that reflects your user base. If you're not familiar with confidence intervals and margins of error, use a sample size calculator to determine appropriate sample sizes.

• Timeframe: Run experiments for long enough to gather meaningful data but not so long that the experiment becomes outdated.

• Statistical Significance: Statistical significance is a fancy way of saying that your data is reliable—it's not just random noise. Again, if you're not familiar with statistics, you can use an online calculator to check for statistical significance.

Support in-depth product experimentation with UXCam

Product experimentation is only effective when you have an abundance of high-quality data to work with. You need it to form sound hypotheses and to test those hypotheses.

UXCam makes it easy to collect and analyze data for product experimentation. Session replays let you zoom in on user behavior, while segmentation tools help you quickly filter data. That way, you can focus on the information that matters most—for example, patterns in behavior associated with user success or failure—and make more informed decisions.

Ready to do more with your product experimentation? Sign up for a free trial of UXCam today.