How to Run User Surveys in the Age of Product-Led Growth

How to Run User Surveys in the Age of Product-Led Growth

We're running a a webinar on implementing microsurveys to boost response rate and product adoption. Register below to join us. 👇

How to Run User Surveys in the Age of Product-Led Growth

How user research is at the heart of product management

The beginnings of the discipline of product management are said to have started at P&G.

In 1931, a memo outlined the need to hire people to oversee and manage all aspects of a brand.

This memo emphasized field testing and customer interactions and was a precursor to “the Hewlett-Packard Way”.

The HP Way is a policy that made product managers represent the voice of the customer and built customer feedback into every team's process and philosophy:

Originally understanding customer (and market) needs to be able to correctly position and package the product was a role within the Marketing function. However as software industries evolved, Product Management came into its own, and became responsible for a product meeting demand, while Marketing became responsible for acquisition, brand and messaging.

In the early days of software product management, the product development process was modeled on manufacturing and process industries; linear and sequential. The Waterfall model set clear stages, deadlines and activities to enable successful delivery.

The first and most important step was to establish the requirements for the product, including gathering needs from end-users, through consultations, validating these identified themes and translating them into a specifications document. Product managers responsible for the Waterfall process were also responsible for this “Requirement Gathering and Documentation” phase.

Modern product management has left user research behind

In 2001, in the mountains of Utah, seventeen people met to discuss some of the problems associated with the current product development processes; where excessive planning and documentation contributed to products that were out-of-touch with users. They penned the “Agile Manifesto”, in which they articulated some key principles to help unite the various development frameworks (e.g. SCRUM) that were in use at the time:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Although their thinking was not novel per se (e.g. other companies such as Toyota had innovated with concepts such as “Kanban” (signboard system), “Kaizen” (continuous improvement) and “Genchi Genbutsu” (“go and see for yourself”) that are still relevant today) their manifesto is the basis of many modern product management techniques and approaches.

Agile as a culture increased collaboration between engineers and product managers, and helped remove some of the existing silos between research and development. The Lean Startup methodology has furthered some of these concepts, and now Product Management is a serious and possibly the most important function at a software company.

Ship it?

All of these innovations in the product process are about building more successful software, more quickly, and more efficiently. Because of the inefficiencies in how software was previously delivered, many of the improvements that companies and teams have focussed on are around shipping.

Many tech companies evangelized shipping quickly; Github popularized the ship it squirrel from it’s 2012 blog post where it explained the need for a mascot and emoji. Facebook’s early motto of “Move fast and break things” was centered around developing software more quickly, even if it meant breaking the end-user experience. Hackathons (including “Shipit”, Atlassian’s unique name and take) and Continuous Delivery have become the standard at all modern software or technology companies.

However in this frenzied rush to build software, one key element has been left-out: user research. Writing code is much quicker than arranging and conducting user interviews, so the latter are often left out or held back for larger projects.

User research has become a component of the design process, and accountability is left to the designer, resembling how research was incorporated during the Waterfall days.

Continuous User Feedback is the other half of a beloved product

Teresa Torres, a renowned product thinker and writer, references “continuous product discovery” at a talk she gave at Productized back in 2017, as a change in approach such that product teams are doing research every week. She says:

“most companies overemphasize delivery, and underemphasize discovery”.

This is a common issue amongst teams, because building and shipping product continuously is much easier than continuously collecting user feedback. And although teams now more effectively employ product data in making decisions, this often doesn’t answer the question of “why”.

That’s led to tools such as FullStory, that provide a visual representation of what users are doing and struggling with, but even these don’t sufficiently help product teams understand their users.

This is especially important because on average only 4% of dissatisfied users take the time and effort to get in touch with a business (which means that there are ~26 other unhappy people for every person that complains).

To best employ the lean approach towards product, we need Continuous User Feedback, akin to continuous deployment. This means user feedback should be collected throughout the user journey and product lifecycle, not just in the beginning design phase, or on a per-project basis.

And some teams have already made the shift towards doing this; leverage key lifecycle events to trigger user surveys, or sending on a regular, recurring basis, like in the example below.

💡 Continuous User Feedback requires a combination of (small) user surveys and feedback points inside and outside the product. Some examples of ways that this can be done:

• A clear way for users to leave feedback on their product experience at any time
• Upon release of new features or changes to the interface; either upon activation or failure, asking users how valuable the change was
• On key pages / workflows to understand user satisfaction
• On parts of the product that might be under consideration for improvement or changes
• During onboarding and setup to understand goals and motivations (or even “aha moments”), which may differ by persona

As we move into a product-first world, more of this research and feedback collection needs to be handled at scale, and can’t rely on the analog methods such as one-to-one interviews or usability studies alone. Even the experts (user research professionals) can confirm that current methods aren’t enough; 75% of respondents to the “State of User Research” study said their companies didn’t do enough user research.

Today’s methods for incorporating user feedback and conducting product research are not fit-for-purpose

Conscientious product leaders will encourage their PMs to “get out of the building” (as Steve Blank was once yelled to do) and many product teams try to continually gather user feedback… but it’s hard. Response rates for surveys are low, interviews are time-consuming, and soliciting feedback from the right people requires more data and engineering work than teams can dedicate.

In search of speed, some teams now employ the week-long Sprint methodology, first developed by GV (previously Google Ventures). This is about solving problems quickly, through rapid design and iteration, and relies on actual user feedback. Nevertheless this is primarily employed during the early, design phase of a new product / feature, and finding participants for the feedback is still one of the most challenging components.

Today, most companies leverage user surveys, and below is a common example of this; user survey sent via email, linking to an online survey (Typeform), triggered after a key point in the user journey (in this instance, cancellation of the subscription).

And while most people definitely prefer to respond to a survey via on online form (91% of respondents in a SurveyMonkey report) compared to responding via email (only 28%), this doesn’t reflect the wide variety of UX that’s in use.

For example, the DocuSign online survey below, feels like it’s from the 90s with little regard for how easy it is to complete (a short text field for an open-ended question like that… seriously?)

While most companies have adopted online surveys and a corresponding SaaS tool, most of these are from the era of a marketing-first world, where the website was the primary location for online collaboration.

Unfortunately for most lifecycle-triggered email surveys, the problem is that they are too late. As a user I have switched context by the time I arrive in my email inbox and see far more pressing tasks than responding to a survey.

In the Geekbot cancellation survey example above, I am no longer thinking about Geekbot, so expecting me to go back and remember my thoughts and articulate these requires a higher cognitive load than I have capacity for. There is also little motivation to undertake a multi-question survey for a product I’m no longer planning on using.

Imagine if this survey came right after cancellation, while I was still in the product interface; the chances I would give feedback would be much higher. This is exactly the approach that the newest cutting-edge SaaS products are employing. To understand the future of user surveys we need to understand what these product-first companies are doing.

Two products that demonstrate the new approach are Stripe and Segment. They both offer users the chance to give their thoughts inside the product, when they are actively engaged, feedback is fresh and motivation is high.

Stripe clearly invites feedback at any stage of product use through it’s ever-present CTA at the top of the page.

Here, Segment, offers a banner to customers on a particular page, asking for suggestions to improve this page. This might be a precursor to planned improvements and is a great way to solicit user pain points.

Having clear prompts to let users provide feedback are key; most users will not end up giving feedback unless it’s easy to do so and without a clear prompt / request. This is because creating user behaviors requires motivation, ability and prompts, as we explain in this post.

In another example, Booking.com prompts a user to provide feedback after successfully completing a key workflow: booking accommodation. Capturing user feedback and sentiment at key points in the user journey (at both success and failure points) is another smart way of introducing user surveys and regular feedback into your product.

The only unfortunate aspect is that Booking.com may have gone overboard by displaying two separate user feedback prompts (NPS and “How are we doing?”) amidst a deluge of CTAs.

In the examples above, feedback collection has been built into the core product interface, and this requires engineering efforts in both the initial build and on-going changes and maintenance. As more tools become available to handle dynamic product experiences without coding (such as Chameleon) this feedback collection can be designed, built and deployed without engineering.

New technology means user surveys can now be targeted based on user data (such as attributes and actions), added in context (in the product, at the right place) and linked to other product data, to provide a holistic understanding of user intentions, motivations and confusion, to complement other data.

Giving product teams the freedom to design, deploy, edit and analyze these in-product surveys relieves the engineering team to focus on more sophisticated problems, and empowers product to re-take responsibility for being close to the customer. In-product surveys also lead to higher response rates and more accurate feedback, and are preferred over email surveys.

The real opportunity to drive more “product success” and product-led growth lies in adoption of a Continuous Feedback approach, and new technology is making adoption of this approach much easier than needing mass culture change throughout an organization.

⚡ Microsurveys are better than the traditional user survey

Yet simply more user surveys inside the product are not sufficient. Survey abandonment is common, and teams have to find a delicate balance between number and depth of responses to collect sufficient data. All the best practices around user surveys today suggest that surveys should be short. But how short? 5 questions? 10 questions? How long should it take a user to fill out a “short” survey?

We think the new standard for in-product user surveys will be “microsurveys” -- ONE QUESTION with space for text comments and feedback. A major reason Net Promoter Score (NPS) surveys have grown in popularity so much is the standardized format; a single rating on loyalty followed by optional text comments.

In the example below, HubSpot makes it stupidly clear what it wants to know.

I’m in flow and the cognitive load to respond to this user survey is delightfully low. I’m motivated to provide feedback because I’m still using the product, and the question makes me feel valued; while the format respects my time.

We’ve applied a similar approach to enable product teams to create microsurveys inside their product, without needing engineering. You can now create Chameleon Microsurveys with a rating or button answer and text feedback. These can be launched from banners, icons, buttons, modals etc. just like Stripe and Segment do. They can be shown only to particular users based on who they are or what they did, or based on the page and it’s content. And all the data flows seamlessly back to your analytics tool (e.g. Mixpanel or Amplitude) so you can join together qualitative user feedback alongside product usage data.

We know that response rates for an in-app survey are roughly double that of email, with Chameleon customers such as SkySlope and Mixpanel, improving their ability to collect quick user feedback. If you’d like to get a demo and discuss which microsurveys should be incorporated into your product then just email us.

In-product microsurveys allow users to express their opinions with low cognitive overhead and this is an important part of the user experience. We’ve all known how frustrating it can be when you want to express dissatisfaction at something and have to waste time figuring out how to even get in touch with the company!

Bring on product-led growth with continuous feedback through in-product microsurveys

Product-led growth can be amongst the most effective and efficient strategies in building a successful product and company. It also fits the trend for the modern user that wants self-service and the ability to interact asynchronously, at their own convenience.

Building and shipping quickly is an important part of creating product-market fit (a necessity for product-led growth).

Usage analytics and A/B testing are key sources of truth in whether functionality meets market needs and expectations.

Yet, low adoption of new features continues to be a major problem plaguing product teams around the world.

The solution isn’t necessarily running more tests or moving more quickly, but actually incorporating continuous feedback and product discovery into your product methodology and process.

Within the next few years every product will offer in-product user feedback options as standard. But right now we are on the cusp of this change, and teams that adopt this new technology more quickly will be at a significant advantage.

Thanks to Martin Eriksson and his great article on the history of product management for Mind the Product here.

P.S., you can build, target and edit microsurveys with Chameleon.