Designing Surveys
2015-04-05
Here's a guide to help you design and run a survey project adapted from the workflow I use at GitHub.
Even though we tend to have short surveys (~10 questions), designing those same surveys doesn't happen quickly.
Step 1. Open an issue and propose goals
We write GitHub surveys collaboratively. At the beginning of a project, the partnering team (e.g., @github/product-management) opens an issue in the UXR repository with:
Goals – What do you want to learn or validate?
Audience – Who should receive this survey? (e.g., active users, members, administrators)
Timeframe – When does the survey need to ship to be most helpful?
Responsible person – Who will be responsible for helping to ship the survey project?
Example goals:
Learn about feature awareness
Establish product satisfaction baseline
Develop archetypes or personas to understand the needs of different audiences better
Understand behaviors (e.g., where do you store large files outside of GitHub?)
Step 2. Kick-off meeting
Face-time is a critical component to starting a research project, and the kickoff meeting:
Establishes the project timeline.
Determines roles and responsibilities.
Provides the partnering team a chance to share more context and demo the product.
Gives uxr folks a chance to discuss the research process and answer questions.
Finalizes research goals.
Step 3. Collaborate on questions and approach
Draft questions. A @github/uxr team member will draft an initial list of questions as a Pull Request. Over the next several days, the partnering team will work with uxr together on edits, share feedback with comments, and make suggestions. The result is a markdown document that we all feel confident about that gets merged into the UXR repository.
Set up survey tool. Then, a @github/uxr team member moves the content to our survey tool, where we'll conduct another internal review of the questions and answer choices and validate that the interactions (e.g., logic-based flows) work as intended.
User recruitment. We'll also work together on a plan for how to distribute the survey to our users (e.g., via email or in-app) and partner with the @github/analytics team to identify the SQL query needed to pull the user list.
3rd party tools. We use third-party tools that talk to one another to ship surveys: SurveyGizmo, Campaign Monitor, and Google Analytics.
The result will include the following assets:
Markdown file
Survey page
Copy for in-app popover and flow (if needed)
Survey email (if needed)
SQL query
Note: We always include a question at the end of each survey designed to recruit participants for future interviews and pre-releases. This is one of our best sources for qualitative follow-up.
Step 4. Survey the right people
We have clear research goals and great questions, so we must get the survey out to the right people!
When recruiting for surveys, we must go beyond what is easy (sharing in our social network - aka convenience sampling) to reduce bias by sampling techniques that reach and engage people from different backgrounds (account age, activity, geography, etc.). Superfans are reliable and willing participants, but we can learn more about our blindspots from people we don't know and those who have stopped using or rarely use our products.
Understanding bias:
Any survey that depends on individuals who self-select into the sample (e.g., voluntary survey) will capture only the views of those who try to voice their opinions. This group typically includes people who feel strongly about an experience and those who have a bit of free time on their hands. Neither is likely to be fully representative of the entire population.
Surveys are subject to response bias and errors in self-reporting (people share variations of the truth). To better analyze survey data, projects should be able to map answers to actual user account data when possible.
Techniques to mitigate bias:
When designing a survey, the research team uses the following techniques to reduce bias and measurement errors:
Randomization helps to reduce bias. We use randomization to help reach a relevant population distribution and shuffle closed answer choices.
During analysis, we take raw survey response data and map it back to account information when available (e.g., account age, membership to paid/open source org, individual paid/open source plan, geography).
We then look at the over/under-representation of particular populations. Sometimes, we can come up with theories for why there were groups of non-responders (e.g., the survey is in English, which may be why there's overrepresentation in English-speaking locales).
Closed answer choices. Our current survey tool (SurveyGizmo) has known bugs in question randomization. However, to help reduce bias, closed-answer choices are usually randomized. Closed questions provide a limited choice of answers from a pre-determined list. Open questions invite participants to share answers in their own words (good qualitative insights).
Vocabulary ambiguity. When designing questions, we do our best to avoid vocabulary ambiguity. If people don't understand your question in the same way and can't respond equally easily, you will get a measurement error.
Double-barreled. When designing questions, we avoid asking two questions (e.g., How satisfied are you with GitHub Issues and Pull Requests?). The user may feel poorly about GitHub Issues but is very satisfied with Pull Requests. However, since it’s just one question, they’re forced to answer only in one way so that you will have a measurement error.
Note: Recommend reading Writing Surveys that Work, by Rebecca Weiss (~5 minutes)
Step 5. Pilot and pre-test
When we ship a survey, we begin with a tiny % of the population and schedule time to review early answers that may show that we need to fix things before shipping to the entire population.
During the pre-test review, we look for:
Bugs – Are questions being skipped or missed? Are the buttons working?
Vocabulary – Do all respondents appear to interpret the question and terms (e.g., "Agile methodology") similarly?
Content – Are some people writing the same response in an open-ended field? Should that response be a closed-answer choice? Did we forget anything?
UserIDs – Is the data between systems connecting as intended?
The data collected in your pilot will clarify where your survey needs refinement. If there are big changes, the pre-test data must be disqualified. Being prudent and sending the pre-test to enough people can stress-test the clarity of the questions and data you’re gathering while considering how much data you can comfortably throw out.
We typically pause a survey, quickly iterate to improve on areas we overlooked (e.g. add a closed answer option or change copy), disqualify early answers, and then ship to the rest of the segmented population.
Step 6. Ship and monitor
After the pre-test, early review, and iterating, we're ready to ship the survey to the segmented population (unless we need a second pre-test).
It's essential to check on the survey daily, look for inconsistencies in the UI (provided by a 3rd party that sometimes breaks), and check engagement with GA event tracking. If participation is low, you may learn that you need to increase the % of people receiving the survey.
Step 7. Analyze and report
An assigned partner on the @github/uxr team lets the feature team know when we have enough data to begin an analysis. During the analysis phase, we'll work together to surface insights and produce a report, following a template that organizes information:
Abstract
Background
Observations
Recommendations
Aspirations
Analysis and draft reports result in a lot of lively discussion and many edits. Good data encourages us to look at survey results in many different ways.
We go beyond what the survey tool tells us is the pure outcome and instead map answers to personas, tenure, account type, and age, and even look for trends in how particular types of users answered multiple questions.
