Designing a Foundational Study
2015-01-10
When products experience major change, ongoing iterations, and growth with newcomers, it's vital to stay in touch with user behaviors by conducting annual/bi-annual foundational studies.
A foundational study has three parts:
Quantitative research – deep dive into what's happening by the numbers and segments.
Survey research – gather demographic, feature feedback, and recruit interview participants.
Qualitative research – field interviews.
Past and current behaviors are an excellent proxy for future behaviors. A foundational study can keep you and your organization in touch with the experiences of new and older cohorts. You'll discover behaviors that don't (but should) exist, which helps to inform product development priorities (quality improvements and innovations).
Research goals
Before embarking on a foundational study, set your goals. Here are some for a project we're working on for GitHub Organizations (our permission or Role-Based Access Control [RBAC] system):
Establish baseline metrics. This will help you to measure the impact of change(s).
Find the story. Learn how people are using your product, problems they are trying to solve, areas of friction that they encounter most frequently.
Develop archetypes or personas. New users, established users, paying users, free users. Write down their motivators, beliefs, attitudes, needs, and use cases (e.g., What is life like for a newcomer? What has changed for more established users?).
Connect with a cohort. This is a group of people you can stay in touch with and learn from as part of a longitudinal research effort. These will be people you check in with regularly (e.g. quarterly) and might also make wonderful beta candidates.
Methodology
The guiding goal for our approach is to dig deep, amass a diverse data set, and then set about finding the story ...
Step 1. Gather baseline metrics
Work closely with your analytics/data teams to get a baseline on a set of metrics for the project that include a mix of new and older user behaviors.
If revenue is important, make sure to learn more about tracking month recurring revenue (MRR), annual recurring revenue (ARR), and possibly net promoter score (NPS).
Step 2. Conduct a survey
Conduct a survey(s) to:
Validate assumptions.
Uncover areas for product improvement and prioritization.
Gather demographic data that you would otherwise not have access to.
Recruit interview participants for usability tests, betas, and follow-up.
Step 3. Set up interviews
It can be really hard to interpret quantitative data if you haven't been in the field looking at what's going on; watching and listening to people can give you insights and help inform future quantitative studies.
Go outside of your bubble, there's nothing like talking to people who aren't like you or your friends who are using your product:
New users – This group should have less than 45-days with your product. Surface the mental model newcomers bring to the product and surface nuances of expectation and friction getting started.
Established users – This group should have 1+ years of experience with your product. We'll learn about the mental model created by the product and surface nuances in workflow (what works/doesn't and why, how are people augmenting your product's shortcomings?).
Step 4. Organize and share findings
Go back to your hypotheses, how did things line up? What surprised you the most?
