Securing Success of an Open Beta With Mixed Methods User Research
This is an article format of the talk I gave for UXDX Community in 2023. See the full session with other speakers here.
In the product development lifecycle, startups and small businesses often find themselves in a position where they need to run an open beta of their product. This involves releasing a beta version of a product and letting real world users test it out.
If you’re working as a product professional (no exclusions here: this applies to product marketers, product owners and managers, as well as product designers), and you’re given the opportunity to be a part of an open beta, there are research and observation activities you can perform during the course of the beta to ensure that your organization finds the answers it needs to make decisions relevant to business goals.
Defining “Beta”
For the context of this article, we can define beta as when an organization has a releasable version, ( v 0.01 … ), of a new feature or product that it wants to get in the hands of real users. The organization could have many goals that it wants acheive by running a beta program, an important goal will most likely be finding and fixing bugs so that they can follow it with an initial stable release, ( v 1.00 … ).
The time period of a beta can also be a huge opportunity to product professionals at the organization to deliver continuous insights through user research. By reaching out to your users during the beta you can uncover actionable insights that can have an impact on future product improvements and usability enhancements.
If your organization is large enough to have dedicated product marketing and sales professionals, the insights found during a beta can have an even larger impact outside of product and engineering teams, and can aid marketing and sales in crafting engaging stories for prospective users and customers.
Open vs. closed beta
Your organization’s product and specific circumstances will help to determine whether an open or closed beta is appropriate. For example, if your organization does government contracting and is building a sensitive product for users with very specific goals, it will make more sense to run a closed beta.
Open beta:
- Anyone can participate
- Potentially larger number of participants
- Open feedback loops (public)
Closed beta:
- Invitation only
- Smaller number of participants
- Closed feedback loops (private)
Regardless of whether your organization chooses an open or closed beta, you can still pick the relevant user research methods from your toolkit and tailor them to your specific circumstances.
Phase 1 — Discovery
Prerequisites
There are some things you’re going to need ironed out before you get started running user research during a beta. If you’re lucky enough to have been involved in early discussions about the product with leaders and teammates, you and your team will likely have formed a hypothesis about the product concept. Your team’s hypothesis should be collaborated on and document the problem your product aims to solve, the value proposition of the product, and general understanding of your target users including demographic information, as well as industry information if the product is B2B.
Goals
- Get to know your stakeholders
- Validate product idea and value proposition
- Get to know your target users
- Define or validate early feature concepts and terminology
- Define timelines and milestones
Early discovery research is also key in validating the product idea and value proposition. We don’t live in a perfect, linear world, so there may be some overlap in when you perform concept validation research. You may find yourself in a situation where an engineering team has already started to build a solution for a problem that hasn’t been validated with your own reasearch. If you find yourself in such a situation, don’t panic, you still have valuable time before the beta is released to dig into research mixed methods.
Get to know your stakeholders
One of the first steps you should take during beta research is to do a round robin of your organization and get to know different stakeholders involved in the project. The entire organization has a goal, but the stakeholders within the project may have different things they aim to get out of the beta, so getting to know them and what their specific role needs to be successful is key in building relationships that empower the entire organization.
Once you’ve gathered relevant stakeholders and understand what each can gain from a research partnership, you can begin to lay out a timeline and milestones for the course of the beta leading up to the beta exit or v 0.01 …
User research is something that is done in service of others and towards a greater goal — for the organization, as well as for customers and end users.
It’s rarely productive to perform user research solo, or without the input of others. Avoid the trap of design team silos and include team members from other departments as early in your process as possible. This will help to prevent your research from being ignored and will help you keep focus on serving the needs of the organization.
A great practice is to form a core team of stakeholders that will be involved in user research over the course of the beta, and it is often helpful to keep a list of “informed” stakeholders — likely people in leadership positions who will not be involved in day-to-day research practices but need to stay informed of progress. Forming a core team will also allow you to share duties of research among a multidisciplinary group, so each team member can play to their strengths as well as share the workload. Having a core team with members from different departments can also help disseminate research results further throughout your organization and increase the impact on other teams that benefit from research insights.
If you are working on a product as a UX designer, your priority is most likely usability of the product. If you were to work alone on research you might miss insights you could have uncovered if you were to have included a product marketer, salesperson, who may be focused on overall messaging or core motivations that can make or break a product’s success in the market.
I like to think about it in Lord of the Rings terms — Frodo didn’t make it to Mount Doom alone. He had a fellowship of elves, dwarves, fellow hobbits, and men — all played key parts in defeating Sauron.
You may find yourself with more or less stakeholders to include in beta research depending on your organization’s size or maturity. Planning research for higher complexity organizations with many different departments will require you to be mindful of the different stakeholders involved and their respective goals and timelines related to the beta.
Not all of these groups need to be involved in every aspect of research planning but some stakeholders may be more involved in certain parts of the process. For example, it may not make sense to involve QA and support teams in product discovery or validation, but once you start to do usability testing those teams are going to be more interested in that part of the research as it is more relevant to their goals.
Validate the product idea
User interviews or focus groups
If you have your team’s hypothesis about the value your product can bring to the market you will want to start to validate that and make sure that spending your organization’s engineering resources building a beta. You can do this by talking to prospective customers or users that fit the product’s target demographic. At this point in the process you don’t need prototypes, you can start with just interviews or focus groups. The goal here is to see how people are thinking about the problem that you’re aiming to solve and where your solution can fit in or even find an edge.
Competitive analysis
One of the easiest ways to wrap your head around a product concept or market landscape is to do a competitive analysis of other products that solve a similar problem or are closely related to the product concept. It is one of the most lightweight and powerful tools in the user research toolkit because it doesn’t require a lot of time recruiting participants and it’s something you can do asynchronously with your core team of stakeholders. You can do competitive analysis as desk research and act as a ux-er — focusing solely on tasks, user flows, and usability, or you can approach it from more of a product manager focus and compare feature sets and offerings between products to see where there is a gap to fill.
If your core team of stakeholders includes marketing, you can even approach it as a blended UX and marketing competitive analysis where you take a broader view and look at the overall customer experience — how the competitor products are being positioned and what the marketing and sales strategies are for those products.
Get to know your target users
Once you have performed user interviews and done a light competitive analysis you should have a baseline understanding of who your target users are for the product. For the purposes of the beta, you should start to think through how you plan to recruit beta participants.
As soon as there is a product concept and general hypothesis you can begin to recruit participants. So if you start this process early, the participants in early discovery research can turn into beta participants who continue to participate in any research you do during the beta.
There are a few DIY approaches you can use in recruiting beta participants, as the internet is still a rich place full of different communities that may be your perfect target. If you have budget limitations and more time, you can solicit beta participants in online forums and communities, or even go to in person meetups relevant to your product and speak with people in person.
Unless you work for a large established business with a large existing user base you can tap into, you may go through some tiral and error in enticing participants. If you are limited by time or aren’t having luck finding beta participants you might want to consider using a paid recruitment service that can handle the recruitment for you.
If your company already has a product with an established user base, congratulations. Leveraging any existing user base you have and if your company has a lot of brand loyalty you will probably have an easy time gettining interest. Creating a beta sign-up form and linking it in relevant places where your existing user base can find it is an effective way to build your pool of beta users. It helps to do this early on and keep it active throughout the course of the beta so new participants are able to sign up if engagement starts to drop off over time.
Keep participants engaged
No matter how long or how much you need to test during the beta, you should keep in mind that engagement from beta participants can decrease over time. The shiny new promise of your beta might get less exciting for participants even if they become daily active users, and their continued willingness to participate in things like surveys, interviews, and usability tests might drop off. Have a plan in place for how you will handle this.
It is standard practice to compensate research participants for their time and there are calculators available, like this one from ethn.io, that are a great reference for how much you should plan for in terms of research incentives.
If your product is complex and requires time to set up and for users to see the full benefit, you might consider incentive structures for longer term studies. Be mindful of any potential bias you may create in your results and eliminate potential bias where you can.
Define early feature concepts and terminology
If your product is complex or introduces abstract concepts your team will probably have come up with terminology for features or functionality. The discovery phase of research is the best time to talk to participants about their mental models and expectations about what terms make most sense. Describing a feature to participants and asking them what they would call it is a great way to name a feature or validate that the name you have given it is on or off base.
Set timelines and milestones
Each member in your core team of stakeholders is going to have an ideal timeline of when they need certain results or deliverables. Marketing is going to have a specific timeline of when they need to start communications or press releases about the product, engineering is going to have their own timeline of when they need to resolve critical bugs or usability issues before v1 is released, etc. Sitting down with your core stakeholder team and creating an overarching timeline that makes sense will help guide your efforts and align your team about your beta research plan.
Ideally as a product designer or researcher, you are brought into beta planning processes early. If you don’t have that blessing, take the opportunity to check in with stakeholders as early as possible and get a pulse on their goals and timelines. Scheduling regular check-ins with your core stakeholders is a best practice to ensure continued collaboration, and it can give you the opportunity to adjust your timeline if things change over the course of the beta. Regular check-ins also give you scheduled time to socialize early results as you progress through your overall plan.
Tip: Build dedicated time in your plan to analyze results so that stakeholders know when they can expect reportable findings. Not everyone is fluent in user research processes, so taking time to educate on process helps them understand level-of-effort involved.
Beta exit criteria
What is the end of your plan? When will it all end? You can answer these questions by working with leadership and your stakeholders to define what a successful beta exit looks like and defining criteria for an exit.
Examples of exit criteria:
- Target number of active users is met
- Critical issues have been identified and resolved
- Feedback has been analyzed and resolved
- Usability issues have been analyzed and resolved
Phase 2 — Product definition
If you still have some runway before you beta gets in the hands of your users you can move into product definition. The goals of this phase could be to validate the feature concepts and terminology you defined in phase 1, to determine optimal pricing if your working on a SaaS product, and define any data collection methods you want to use during the beta.
Determining which types of data, quantitative or qualitative, you will focus most of your energy on will depend on your product and specific circumstances. If your product is in a highly sensitive space such as cybersecurity, you might be limited or prohibited from collecting unmoderated analytics data. If this is the case, you will need to work with your core stakeholders to determine key data points you need and collaborate on how you might glean those answers from moderated studies. ATLAS.ti has a great starter resource if you’re considering turning qualitative data into quantifiable results.
Validate feature concepts and terminology
Stakeholders to include:
- Product
- Marketing
- Tech writing / documentation
To get feedback on terminology you’re using in your product you can utilize research methods such as surveys where you describe a feature and have respondents select or enter a term that makes the most sense to them. A good follow-up question in the survey would be an open-ended text response where you prompt participants to explain why they selected their answer. This validation method also works extremely well in a moderated study or interview where you can ask participants to think out loud and elaborate on any other terms they think describe the feature best.
If you happen to be running a usability test, a great way to validate terminology used for an abstract feature would be to include a first-click test. This comes in handy particularly in complicated products where there might be many items in the product’s navigation, or when the feature is aimed at a specific subset of users who have a particular goal in mind or need to complete a specific task related to the feature. For more information on when and why to use click tests, see this article from Sauro, Atkins, and Lewis on MeasuringU.
Validating decisions about terminology at this stage will help stakeholders formulate talking points about the product that are accurate to the mental models of target users.
Determine optimal pricing
Research methods to consider:
- Conjoint analysis
- Pricing sensitivity survey
Stakeholders:
- Product
- Marketing
- Finance
Pricing is often a complicated topic that involves more areas of the business than you may regularly find yourself working with if you are a product designer or user researcher. If your organization is launching the beta with the goal of offering a new standalone product, the main questions you will be able to offer insight on are questions such as “how much do users value the product and feature set?” and/or “what are people willing to pay for this product?”. Though you are likely not the final decision maker on what price point your product will be offered at, you can conduct pricing studies to inform decisions of other departments and leadership at your organization.
Doing pricing research early has the potential to have a great impact on how the product is positioned in the market as well as how sales and marketing teams market and sell that product against competitors. It will also allow other teams at your organization to make informed financial predictions and set expectations about revenue growth with the feature set at launch.
There are several different research methods you can choose from depending on exactly what your product offers or what the early concepts for a pricing plan look like. Two good examples for software pricing research include quantitative survey methods such as a conjoint analysis survey or a pricing sensitivity survey. For example, if you want to see what the optimal price point is for the product and you care less about the feature set, then a pricing sensitivity survey may be more beneficial to conduct. If you’re a team of one, looking at a tool like Conjointly, that offers a variety of different survey solutions for pricing research.
Revisiting any competitive analyses you conducted during the discovery phase to see what price points or structure competitors are offering will also help you see where your product and feature set could stand out to consumers.
Analytics and quantitative data collection
If your team is able to, take time before the beta launch to sit with your stakeholders and determine what quantitative data points it would like to collect, including any key analytic events your core team would like to track.
Activities to consider:
- Stakeholder workshops
- Collaborate ideation
Stakeholders:
- Data analysts
- Product
- Marketing
- Engineering
If you’re lucky enough to be able to use a tool like Pendo, then you will have the benefit of retroactive analytics, where once it is installed everything is automatically collected and you can dig into any events that have been collected without having to burden engineering or data science teams with building custom solutions to track user interactions. If not, then you will need to work with your engineering and data science counterparts to define what events you would like to collect, and specifics related to time (daily events, time elapsed since last event, etc).
Tip: Collecting quantitative data can be extremely powerful at telling you what is happening, but be careful to not jump to quick assumptions about why things are happening. Instead, follow up any questionable data observations with qualitative research methods if your team has questions about the why.
Qualitative data collection
Pairing quantitative and qualitative data collection for this phase of your beta will give your core team a lot of confidence for future decisions about the product, and in addition to moderated research methods you can also run some unmoderated studies in the form of feedback forms and open-ended surveys.
Methods to consider:
- Online forum for beta participants
- Feedback forms/ surveys
- Diary studies
Stakeholders:
- Product
- Engineering
- Marketing
- Support
Creating an online forum specific to your beta will give participants a place to discuss the product online and can become a rich resource for your core team to see new perspectives. Additionally, if you have access to any customer support tools then by all means use them. Support tickets can be an amazing resource to find patterns in what users are experiencing.
Phase 3 — Beta launch
Now that your team has progressed through discovery and definition, you should have a clearer picture of any key usability issues you can resolve as. a product professional and how your team is going to position your product in the market in preparation for a v1 launch.
At this stage you have put your product in the hands of actual users and you can now move into the monitoring phase. Based on the processes and timeline your core team established in the previous phases you should be able to continue data collection and monitoring, as well as regularly socializing any findings with your organization.
Activities to consider:
- Round table discussions with stakeholders
- Weekly stakeholder check-ins
Stakeholders:
- Product
- Marketing
- Engineering
- Support
As time passes and more participants use the product you will have more usage data to dig into. What your core team observes in the data can inform the focus of any check-in interviews you may want to conduct with beta participants.
Identify new features to add value
By this point you should have multiple data streams from which you can pick out patterns, themes, or observations about how participants are interacting with your product. Whether you are doing regular user interviews or just reviewing support cases, patterns will likely emerge about how your product stand up to user needs.
Are there common concerns being voiced? Common pitfalls or pain points? Have multiple participants mentioned any gaps in what the product offers? Working with your core stakeholder team to answer these questions will help steer you in the direction of additional features your product could add to drive value for users.
Phase 4 — Post beta
At this phase your team has determined you are near enough to meet the exit criteria that you can transition to the v1 launch phase, or general availability.
Goals:
- Identify areas for future improvement
- Assess usability of the product (benchmark)
- Prioritize post-beta features
If your core team has observed that a certain feature is underutilized, or being utilized less than was expected, now is the time to discuss and prioritize any solutions.
If all critical bugs have been resolved then it is the perfect time to run a usability benchmark for the product. Depending on your product and circumstances, you may consider doing either an unmoderated or moderated benchmarking study. Your choice may depend on timelines or the level of access you have to users.
Usability benchmark metrics you might want to collect could include time on task, task success, task ease (SEQ), user confidence, and overall system usability score (SUS). You could run eith a moderated or unmoderated benchmark depending on your circumstances, and in addition you may also consider a retrospective benchmark survey.
Usability benchmarking is extremely beneficial for any product team in the long run because as you continue to come up with product improvements and new features you can use data from previous benchmark studies to determine whether you are improving the usability of your product with any new feature or enhancement. It gives your organization confidence that it is making the right decisions and reduces overall risk in making changes to your product over time.
Prioritize post-beta features
Based on all the data you have collected during the beta, your team will likely have a laundry list of features in the backlog that could drive value for users. However, you may not have a good sense of what the priority of those features are. Take some effort to prioritize this list of features so that your engineering teams can focus on those most critical to the long term success of the product.
Research methods to consider:
Stakeholder workshops to consider:
Prioritization may include different factors such as what could your team build the fastest, or what things are the most important to users. Have discussions with your core team about any tradeoffs, if needed.
Key takeaways
- Stakeholders = key collaborators
- Use stakeholder timelines as constraints and get creative with planning
- Make a plan to keep beta participants engaged
- Report on insights frequently
- Usability benchmarks will serve you well in the long run, don’t pass up the opportunity
- Balance collecting qualitative and quantitative data when you can
- Have fun!