Designing Surveys

Posted by Chrissie Brodigan on April 5, 2015

Here’s guide to help you design and run a survey project adapated from the workflow I use at GitHub.

Even though we tend to have short surveys (~10 questions), designing those same surveys doesn’t happen quickly.

  1. Open and issue and propose goals
  2. Kickoff meeting
  3. Survey the right people
  4. Pre-test
  5. Ship & monitor
  6. Analyze & report

Step 1. Open an issue and propose goals

We write GitHub surveys collaboratively. At the beginning of a project, the partnering team (e.g.@github/product-management) opens an issue in the UXR repository with:

  • Goals – What do you want to learn or validate?
  • Audience – Who should receive this survey? (e.g. active users, members, administrators)
  • Timeframe – When does the survey need to ship to be most useful?
  • Reponsible person – Who will be responsible for helping to ship the survey project?

Example goals:

  • Learn about feature awareness
  • Establish product satisfaction baseline
  • Develop personas to better understand the needs of different audiences
  • Understand behaviors (e.g. where do you store large files outside of GitHub?)

Step 2. Kick-off meeting

Face-time is a critical component to starting off a research project, and the kickoff meeting:

  • Establishes the project timeline.
  • Determines roles and responsibilities.
  • Provides the partnering team a chance to share more context and demo the product.
  • Gives uxr folks a chance to talk through the research process and answer questions.
  • Finalizes research goals.

Step 3. Collaborate on questions and approach

Draft questions. A member of @github/uxr team will draft up an initial list of questions as a Pull Request. Over the next several days, the partnering team will work with uxr together on edits, share feedback with comments, and make suggestions. The result is a markdown document that we all feel confident about that gets merged into the UXR repository.

Set up survey tool. Then, a member of @github/uxr team moves the content over to our survey tool, where we’ll conduct another internal review of the questions, answer choices, and validate that the interactions (e.g. logic-based flows) work as intended.

User recruitment. We’ll also work together on a plan for how to distribute the survey to our users (e.g. via email or in-app), and partner with @github/analytics team to identify the SQL query needed to pull the user list.

3rd party tools. We use a combination of third party tools that talk to one another to ship surveys: SurveyGizmo, Campaign Monitor, Google Analytics.

The end result will include the following assets:

  • Markdown file
  • Survey page
  • Copy for in-app popover and flow (if needed)
  • Survey email (if needed)
  • SQL query

Note: We always include a question at the end of each survey designed to recruit participants for future interviews and pre-releases. This is one of our best sources for qualitative follow-up.

Step 3. Survey the right people

At this point, we have clear research goals and great questions, so we need to get the survey out to right people!

When recruiting for surveys, we must go beyond what is easy (sharing in our own social network) in order to reduce bias by sampling techniques that reach and engage people from different backgrounds (account age, activity, geography, etc.). Superfans are reliable and willing participants, but we can learn more about our blindspots from people we don’t know and those who have stopped using or rarely use our products.

Understanding bias:

Any survey that depends on individuals who select into the sample (e.g. voluntary survey), will capture only the views of those make the effort to voice their opinions. This group typically includes people who feel strongly about an experience as well as those who have a bit of free time on their hands. Neither is likely to be fully representative of the entire population.

Surveys are subject to response bias and errors in self-reporting (people share variations of the truth). To better analyze survey data, when possible projects should be able to map answers to actual user account data.

Techniques to mitigate bias:

When designing a survey the research team uses the following techniques to reduce bias and measurement errors:

  • Randomization helps to reduce bias. We use randomization to help reach a relevant distribution of the population and to shuffle closed answer choices.

    During analysis we take raw survey response data and map it back to account information when available (e.g. account age, membership to paid/open source org, individual paid/open source plan, geography).

    We then look at the over/under-representation of particular populations. Sometimes we can come up with theories for why there were groups of non-responders (e.g. the survey is in English, which may be why there’s overrepresentation in English-speaking locales).

  • Closed answer choices – Our current survey tool (SurveyGizmo) has known bugs in question randomization, however to help reduce bias closed answer choices are usually randomized. Closed questions provide a limited choice of answers from a pre-determined list. Open questions invite participants to share answers in their own words (good qualitative insights).

  • Vocabulary ambiguity – When designing questions we do our best to avoid vocabulary ambiguity. If people don’t understand your question in the exact same way and can’t respond equally easily, you will get a measurement error.

  • Double-barreled When designing questions we avoid asking two questions in one (e.g. How satisfied are you with GitHub Issues and Pull Requests?). The user may feel poorly about GitHub Issues, but very satisfied with Pull Requests. However since it’s just one question, she’s forced to only answer in one way, so you will have a measurement error.

Note: Recommend reading Writing Surveys that Work, by Rebecca Weiss (~5 minutes)

Step 4. Pilot and pre-test

When we ship a survey, we begin with a very small % of the population and schedule time to review early answers that may show us that we need to fix things before shipping to the full population.

During the pre-test review we look for:

  • Bugs – Are questions being skipped or missed? are the buttons working?
  • Vocabulary – Do all respondents appear to be interpreting the question and terms (e.g. “Agile methodology”) in the same way?
  • Content – Are some people writing in the same response in an open-ended field? Should that response be a closed answer choice? Did we forget anything?
  • UserIDs – Is the data between systems connecting as intended?

The data collected in your pilot will make plain where your survey needs refinement. If there are big changes, then the pre-test data will need to be disqualified. Be prudent and send the pre-test to enough people can really stress test the clarity of the questions and data you’re gathering, while considering how much data can you comfortably throw out.

It’s typical for us to pause a survey, quickly iterate to improve on areas we overlooked (e.g. add a closed answer option or change copy), disqualify early answers, and then ship to the rest of the segmented population.

Step 5. Ship and monitor

After the pre-test, early review, and iterating, we’re ready to ship the survey to the segmented population (unless we need a second pre-test).

It’s important to check on the survey daily, looking for inconsistencies in the UI (provided by a 3rd party that sometimes breaks) and checking engagement with GA event tracking. If participation is low, you may learn that you need to increase the % of people receiving the survey.

Step 6. Analyze and report

An assigned partner on the @github/uxr team let’s the feature team know when we have enough data to begin an analysis. During the analysis phase we’ll work together to surface insights and produce a report, following a template that organizes information:

  • Abstract
  • Background
  • Observations
  • Recommendations
  • Aspirations

Analysis and draft reports result in a lot of lively discussion and many edits. Good data encourages us to look at survey results in many different ways.

We go beyond what the survey tool tells us is the pure outcome and instead map answers to personas, tenure, account type and age, and even look for trends in the way that particular types of users answered multiple questions.


Photo credit: Library of Congress