Running a successful trial and POC

When evaluating the product during the “proof-of-concept” (POC) you must test broadly and deeply to ensure that it meets your requirements and expectations. Here we’ll give you the knowledge you need to prevent surprises or disappointments afterwards. 

Starting a trial or POC differs from creating an account with the product. Usually, you are free to create an account but must enter a free trial or POC before deploying experiences live and thoroughly testing the platform. In some cases, you may be able to start a trial of the product without engaging the sales team, but in others, you may require your account to be manually provisioned. 

Here we’ll provide an overview of the key stages and steps required to run this trial effectively. As you evaluate please be sure to check the features section to look for items to assess and evaluate. 

Trial Plan

As with our other advice, if you plan and document your trial, it’ll likely proceed most smoothly. You hopefully already have a business case written, an evaluation matrix that you’ll be filling out, and you can complement this with a “Trial Plan” which helps lay out the stages and aspects of the trial. This can include:

• Stakeholders involved (e.g. points of contact from engineering, design, etc.)

• Timeline and deadlines (we cover a typical timeline here)

• Link to your evaluation matrix 

• Key outstanding questions/to-do items

• Useful resources (e.g. links to help docs etc.)

During this planning, you should also list key things you want to test or evaluate during the trial. This can include features and use cases. Getting ahead of this early will save time during the actual trial period. 

You will likely not have the time or resources to drive to actual results (as these will take a few weeks of data gathering followed by an iteration and more data gathering, much like an experiment or growth initiative) so it’s better to focus on the outputs e.g. can you use the tool effectively to create what you want?

Some vendors may offer you a template for this trial plan, or you can create your own. Here’s a downloadable one to start you off 🏃‍♀️

Installation

Once you’ve received written / verbal confirmation that the vendor supports your technology (front-end framework, platforms, SPA, etc.), install the DAP platform into your application. There are a few components to this and you should ensure you’re aligned with your engineering team about these aspects:

Code snippet

This will require you to add a code snippet from the DAP (with your account’s unique token) to all pages in your application where you want any in-app experiences to display.

This will mean that every time a page in your application loads, the code snippet also loads and provides a pathway to load content from your DAP. 

If you’re using a CDP like Twilio Segment or Rudderstack, you may not have to touch any code and can implement the DAP code-free by integrating through those services. Installing in this way will also automatically send user attributes and events to the DAP, so you can skip the remaining Installation sections below. 

Ensure that your engineering team is comfortable installing third-party code into your application. If you already use an event-based analytics tool (like Amplitude, Mixpanel, Heap) or a session replay solution (like FullStory or Hotjar) your team will likely already be comfortable with this. If you have no third-party applications installed then it might be more challenging or time-consuming to get the necessary opt-ins. 

User identification

At its most basic, you’ll need to send a unique ID for each user via code. You should use the same ID within your database and when identifying users in other applications (e.g., a CDP or analytics platform) so that all data about that user is attached to the same profile. 

This will mean that whenever a page of your application is loaded, the DAP will know which user is loading that page. 

Some DAPs provide a way to encode/hash this User ID for further security; this will prevent third parties from accessing the list of User IDs and any other associated data. In Chameleon, this is called “Encode User IDs,” and we highly encourage all teams to implement it. 

As a reminder, if you’re installing via a CDP such as Twilio Segment or Rudderstack, you can skip this section because user identification will happen automatically (you’re almost certainly identifying users to these tools, which will identify them to the DAP.)

User attributes and events

In addition to identifying users, you can also send other data about users: 

In-app experiences are helpful if you can target them to groups of users based on their characteristics or progress, so sending this data to a DAP is essential. 

If you have data privacy concerns, you do not necessarily need to send any personally identifiable information (e.g., email address), although mature DAP solutions typically have good security postures (e.g., SOC 2 Type II and GDPR / CCPA compliance) that allow them to accept sensitive data.

You don’t need to send all attributes you currently track in your database; instead, think about a handful of audiences/segments that would be the first ones you would like to target and identify the properties or data needed to define those.

For example, targeting:

• New users would likely require the created_date property so you can filter users created after a certain date e.g. your trial period

• Users based on their subscription may require a property associated with their plan so you know who is an upsell opportunity

• Admin users would require a property based on their role

Figure out the first 5-10 attributes to send and include that as part of the engineering story/ticket so that your engineers can do the installation, user identification, and data sending all at once, and it doesn’t feel too overwhelming. 

As a reminder, if you’re installing via a CDP such as Twilio Segment or Rudderstack, you can skip this section because user data will automatically be sent to the DAP. If you want to control which data your CDP is sending, then you should be able to do this from the CDP web app, or you may need to make some code changes. 

Company or account-level data

If you are a B2B platform that serves companies/accounts, you may want to send data associated with those companies/accounts (such as plan, LTV, number of users, etc.). This will work similarly to user data (first requiring a Company ID and then sending Company attributes). 

Consider which attributes in your database are attached to individuals and which to accounts, and then specify clearly what company/account data you’d like your engineering team to send. 

Verifying installation

Once you’ve installed the DAP then you should be able to receive data into the DAP platform and be enabled to publish any Experiences live. It should be straightforward to validate and troubleshoot this. Some common mistakes/issues that we see arise include:

❌ Not including the code snippet on all pages

❌ Not identifying users uniquely (using the same ID for all users, or not sending the correct ID)

❌ Not adding an exception to a Content Security Policy (CSP) which in turn blocks the DAP from running

Connecting integrations

Once installed it’s probably a good time to connect any key integrations that you want to exchange data (either to send data to your DAP to enable better targeting of in-app experiences; or to send data your DAP is collecting about user engagement, to further conduct analysis or use in user communications outside of the product, e.g. email). 

Beyond data integrations, there may be other useful integrations, so check the roster and see which tools you might already be using!

Note: In most cases, the integration uses the User ID as the primary way to connect data about the same profile. However, some integrations may require Email as the key (especially if they work with non-authenticated / logged-in users), in which case you’d be required to send Email as a user attribute to the DAP to enable the integration to function fully. 

Setting up styles

Any in-app experiences you create must look on-brand / native and compelling. For this, we strongly recommend involving your design team to help set up templates/themes and your general styling. This can be as basic as fonts and colors, but can also include spacing/padding/margins on components, hover states, curvature, etc. 

Building, deploying, iterating

The next stage is to go build! You should focus on a handful of the most important use cases and try to build at least one “real” experience you’d like to deploy to your end users. Beyond that, we recommend trying to build a variety of experiences to get a sense of the DAP’s functionality and edges, e.g.:

When building, review the “components” available to add to any in-app experience. Instead of trying to build the perfect experience, focus on building a variety to understand what’s possible. For example, try adding different kinds of media, using buttons to start other experiences, playing with triggers/rules, etc. 

Although your job isn’t to stress test or QA the product, this can give you a deeper understanding of what’s possible. You can also ask for advice or guidance from the DAP vendor (the AE or any CSMs or experts available from the vendor) on best practices so you can (a) build up your expertise and (b) evaluate how much coaching and guidance you may receive from your potential future partner.

You can use our “What to look for when evaluating 🔍” checklist to support your evaluation. 

We recommend setting up an internal Slack/Teams channel to gather feedback from colleagues and discuss questions or good practices. You can even start to think about the internal process for going live with a new in-app experience: how would someone request this, who would be responsible for building, what does the review process look like, what testing needs to happen, etc. 

Troubleshooting

You’ll likely encounter some issues or questions: It is a significant technical challenge to ensure that in-app experiences work seamlessly with all kinds of applications, and in some cases, there could be a conflict based on how your application is built, how your data is structured, etc. 

Of course, we’d hope that there are minimal issues and that anything that arises can be resolved quickly or a workaround found, but this also gives you a significant opportunity to test the support and partnership you’ll receive as a customer. 

One of the most common issues we encounter is a prospect not seeing an Experience that they expect to see; this can be for a variety of reasons (e.g. they’ve already seen it; they’re not in the target audience; the display rules don’t match the state of their application; incorrect/incomplete installation etc.) and does requires debugging. The DAP may have a self-serve debugger that can help you more quickly identify what’s going wrong, or you may need to contact their support team to help establish the gap. 

Other examples of issues we’ve seen prospects more commonly encounter include:

• Correctly identifying elements on a page such that in-app experiences attached to these elements continue to show even after changes (e.g. deploys) of the application

• Ensuring data is passing correctly between integrations 

Analyzing results

After your in-app experiences have gone live, you’ll be able to get some data back, which you can use to assess how well you can analyze their impact. 

You should be able to view data in the DAP itself and send it to wherever you collect, store, and analyze the rest of your product data. This might be in your database/data warehouse/analytics tool etc. but we recommend ensuring that the DAP data is another source to your overall understanding of user activity and engagement, rather than being viewed and analyzed in a silo. 

If running any A/B tests, you should also easily understand the difference between the variants and how to choose a winner (even if you haven’t arrived at statistical significance yet). 

Most companies achieve success with in-app experiences through continued iteration and learning so it’s important you feel equipped to be able to understand and learn from whatever you deployed during the trial. 

If you’re running Microsurveys live, you should analyze the results and any comments to assess whether you could use them meaningfully in other use cases. 

Making a decision

You may only trial your top-choice solution, and if you are satisfied, move forward. Conducting full trials is time-consuming, but if possible, we recommend you try two vendors in total, especially if you are buying for the first time. If you’re switching, then you’ll have enough experience and clear comparison points that you likely won’t need to try multiple vendors (or can make the trials very quick).

While you don’t need to make a final decision on your DAP vendor at the end of the trial, we recommend you identify your “product of choice”, i.e. if all the other pieces (terms, pricing, security, etc.) were agreeable/similar, which product would you prefer to use. Knowing that will help you invest time effectively in the rest of the process.  

You likely will have a buying or decision-making committee, and you may need to present your findings, which is why the evaluation matrix and internal Slack/Teams channel can help provide the relevant evidence and documentation to make your case.