6 In-Product Experiment Types for Continuous Discovery (& How to Run Them Effectively)

What comes to mind when you think about experiments? Is it a professor in a lab playing with test tubes and Petri dishes? Maybe it’s creating potions as a kid with the contents of your parents’ bathroom cabinet? Or is it product teams testing feature changes to find the best solution?

All of these answers are valid, but there’s only one we’ll discuss today. Experiments are essential for optimizing your product. They enable you to test new feature designs and ideas to determine their effectiveness. 

Experimentation is a key part of the product development process that shouldn’t be skipped (however tempting jumping ahead to product-wide deployment may be).

So, put down your Petri dish and step away from that cabinet—it’s time to talk product experiments (lab coats optional).

In-product experiments involve continually testing hypotheses in order to validate or reject product development ideas and initiatives. They enable product teams to improve products and features—and ultimately the user experience—with real insights and data on what users want. 

User data and insights are an integral part of in-product experiments. Your initial hypothesis to test should aim to solve a problem you know users have. 

This shouldn’t involve taking a shot in the dark—you should identify real problems through user and product research. Asking questions, testing ideas, and validating initiatives are key for continuous product improvement.

There are often multiple ways to solve a user problem, but you’re always aiming to find the best solution.

Continuous product discovery is the process of consistently researching user needs and UX issues in order to identify problems and experiment with ways to solve them. It is key for making informed product decisions. 

Research involves quantitative methods as well as constantly communicating with users—whether that be via in-product surveys, user interviews, or any other form of user research—in order to find opportunities for improvement, validate ideas, and prioritize solutions. 

This is essential for ensuring your product continues to meet your user needs, while also improving the product for future users.

Effective product experimentation requires you to bake the process into everything your product team does. It involves establishing a culture of product experimentation in your organization. Here’s how.

Constantly researching, hypothesizing, testing, and iterating is an approach to product development that not all companies use. They should—but they don’t.

It might take some effort to ensure the entire team understands the importance of data-backed development—but it’s well worth it. Plus, it doesn’t just stop at your product team’s day-to-day workflows. 

An experimentation mindset is a strong tool on any team, whether it’s customer service, sales, operations, or any other department. Testing new approaches to the same processes will help your team improve the user experience.

Test, test, and test again. That’s how you build a culture of product experimentation: with ample practice. 

Continuously collect data insights on user behavior from product analytics, user feedback, and in-product investigations. 

Data analysis of these insights can help you identify issues to be solved and spot opportunities for improvement.

A product experimentation framework is a must for building a product experimentation mindset amongst your team and company. 

Make sure everyone knows how experiments are conducted in your company—both in terms of the process, the tools used, and the metrics for success.

Following a structured, documented process ensures that all team members, including new hires, are able to get involved. It could be a good idea to create a handbook on your approach to product development that outlines the experimentation method and tools.

So, what makes a great product experiment?

Put simply, it’s the same thing that makes any experiment great: the scientific method.

You likely learned about this in your middle school science classes, but it applies here too. The scientific method is all about observation, measurement, and experimentation.

Like most things when it comes to improving products, experiments start with user research.

User research helps you identify the friction users are encountering, as well as spot low-performing features or pages. Here are a couple of ways you can test the effectiveness of your current designs:

• Microsurveys: Ask users about their experience directly after performing the action you’re asking them about

• Click testing: Run click tests to see where users expect to find things on a webpage

• User interviews: Chat with users about their experience in their product. Do they like the interface? Can they navigate through your site easily?

This helps you hypothesize potential solutions to the problems outlined by users without it being a guessing game.

For the sake of making the explanations easy to follow, we'll include the following example throughout. 

Research could unearth a wide variety of potential optimizations—but you can’t activate them all at once. Prioritize user issues based on impact and user needs in order to be clear about what you’re testing.

However, hypotheses are merely assumptions. You’ve identified the problem, and now you’re speculating about how you could fix it. Testing is where you validate or reject this speculation.

This distinction is key for product development: hypotheses are not facts. Doing user research isn’t enough to fix problems—what you consider a fix, clients may not consider adequate.

Here’s where you clarify what constitutes a successful experiment.

Start by asking these questions:

• What metrics validate your new experiment?

• What are you considering a successful result?

Product experimentation requires collaboration—you want to quickly create effective experiments, and that requires collaboration. 

• Who’s in charge of designing the experiment? 

• Who’s in charge of implementing it? 

• Who will monitor the data to gather insights?

You also need to ascertain what tools you’ll need for the testing process. Where will you collect data? How will you gather continuous research? Make sure you select tools that can help you run and track product experiment tasks and have complete control over your product adoption.

You won’t always effectively validate your hypothesis with one round of experiments. Sometimes you need to test over and over again until you get to the right solution. And, even when you implement changes, you’d want to keep testing to ensure they remain effective.

The product experimentation process never ends—it just restarts, albeit sometimes with different experimentation methods. It helps you make better decisions and provides valuable insights to support product growth.

In-product experiments come in all shapes and sizes, and deciding which is best for your goals is a must. It depends on the feature in question, the stage of development, your users, and more. 

Here are the experiment types you should consider when running product experiments.

A/B testing is a popular way to test new ideas and validate assumptions. It involves simultaneously deploying two versions of a variable—like a webpage, feature, or workflow—to different user segments and seeing which works best for users. 

Multivariate testing is similar to A/B testing, but it involves testing multiple variables at a time. It’s used when your experimentation involves multiple components that you’re rearranging and combining to find the best solution. 

It involves a lot of cross-testing different multiples to find the ideal set-up, but the results are comprehensive and insightful. Here’s a visual created by HubSpot that shows the difference between A/B and multivariate testing.

Usability testing involves testing your product experiments with real users to see if it’s usable and intuitive enough for them to complete their goals.

It consists of giving them a list of tasks to complete in order to observe their process and interactions. Usability testing can help you identify friction points in your product experiment to help validate a new design. 

Click tracking helps you identify where your users are engaging with your product and features by monitoring the actions that users take inside your product. Click tracking, or event tracking, looks at actions like clicks, mouse hovering, form field inputs, forms completed, and more.

Monitoring events can help you understand how the user navigates through your product updates and features, as well as what you can do to eliminate friction points and improve the flows.

This is where creating custom user segments is really helpful. You could then group segments that share the same characteristics for a cohort analysis of their behavior to be more precise in targeting. 

We talked about this in our recent webinar with Userlist on how to use lifecycle segments to retain more users. Watch the webinar recording right here 👇

Fake door testing is a research and experiment technique that helps you gauge user interest in new features or products. It consists of presenting users with a call to action or lo-fi prototype to see how many people will click on it and try to use it.

Painted door testing, as it’s also known, enables you to validate new features and engage interested customers. It can also be useful for prioritizing product updates—if you’re unsure, ask customers with fake door tests and see what’s most popular. 

However, one thing to consider when it comes to fake door testing is that it’s best to use it sparingly. It can cause disappointment with your users and could diminish your credibility within the product if used too frequently.

Beta testing is a form of feature testing that comes further along in the product experiment process. It involves releasing an almost finished version of your new feature in order to assess its overall effectiveness. 

Beta testing assesses your design’s functionality, usability, reliability, and compatibility with real users in order to identify any final issues or changes. 

It is also a great opportunity to nurture your Promoters following an NPS survey. This helps build a stronger client relationship as users feel appreciated and included—privileged to be one of the trusted testees. 

The benefits of this are two-fold: you get eager beta testers and a pool of engaged customers who you can turn to for further user research, like interviews or usability testing. It’s these users that’ll drive product success by being your number-one advocates.

A good experiment is a SaaS product team’s best friend. It takes the guesswork out of development and helps ensure that you’re putting your time and energy into the right solution.

In-product experiments make feature updates, interface redesigns, and product advancement effortless—you can deploy them quickly and easily across your product to get the insights you need. It saves you time and money and ensures you’re giving users exactly what they want instead of what you think they want. 

Remember: Your product development should be based on facts, not theories. Approach experiments with the curiosity that fueled your childhood experiments, and combine it with the knowledge and processes that you’ve developed as a top-notch product manager.