In 2013, I joined GitHub as their first user experience researcher hired to grow our research practice. To date, all of our research projects have been conducted in partnership across teams of product managers, engineers, designers, and writers). You can’t give empathy away.
Here is a high-level roundup of a few of my favorite projects from my past two+ years at GitHub:
Foundational Research
Long-running research studies with many actionable insights falling out along the way:
Understanding project management
Shipping software can be a simple to complex project with all the challenges you can imagine on a technical level with tooling and a social level with communication and process. A lot happens in between both. We started our foundational research with a survey and quantitative analysis of interactions on the site, but it can be really hard to interpret quantitative data if you haven’t been in the field looking at what’s going on.
Part I. Survey: The guiding purpose of the survey project was to help us understand how GitHub users manage software projects, so that we can develop features to better meet their needs. We designed a survey that asked questions about the experiences people have with workflows, tools (including those external to GitHub), and team size. We received responses from about 2,000 randomly selected logged in/out users.
The survey data turned out to be so rich that we presented results in three reports:
-
Process report – Described survey methodology and sample, what we learned about the workflows and methods people use to manage software projects, and identified sources of stress and friction.
-
Tools report – Surfaced insights into users’ experiences using GitHub Issues to manage projects, as well as competitor products like JIRA, Trello, Pivotal, and Asana. The survey asked users about the features they most wanted in a project management tool, and the factors that keep them from using GitHub Issues.
-
Opportunities report– Tied together insights from the Process and Tools reports, and examined the blockers that prevent GitHub users from relying on Issues for project management. We did a deep dive into open text to investigate users’ needs and priorities in their project management systems and identify where GitHub has the greatest opportunities to build on and improve workflows.
Part II. Interviews: Qualitative research seeks out the things that are hard to measure, and watching and listening to people can give you insights and help inform explain patterns from existing quantitative data as well as future quantitative studies.
I sourced candidates for team-attended interviews from our most interesting survey respondents (part 1.). The results of the interviews provided us with greater detail than the quantiative data alone and helped the team decide on the product strategy to pursue.
New user experience on GitHub.com
People who signed up for a GitHub account in 2008 are different than the people signing up for accounts today. In order to help our newest users achieve success, we conducted a foundational study to:
- Learn about the mental models and experiences people bring with regards to version control.
- Discover why people are signing up (go deeper than traffic patterns).
- Surface data about what perceptions people bring to the application (what do people want to do?).
- Ask how we can help people the most (e.g. content recommendations, tutorials, etc.).
To start, members of our product team attended interviews with a diverse set of our newest new users and other people with more than two years of tenure. Our interviews were designed to help us understand how people both learned and taught git and Github. We asked questions to better understand motivations, walked through workflows, and listened to participants excitedly talk about their aspirations.
The outcomes helped us to develop a new user survey, which is presented after sign-up and before the first run. The data from this survey helped us to create a set of predictors useful towards understanding baseline criteria for success and failure, which we then used to inform a set of experiments (noted below), and an exit survey to identify why people go quiet within their first 30 days.
Product billing and payment
In order to sell a product, you need to have a clear understanding of purchasing flows and experiences. I worked with the engineers tasked with refactoring and improving GitHub Enterprise’s purchasing flows (account management, invoicing and upgrading, and self-serve experiences).
This study challenged me in many ways, namely because it required recruiting and interviewing accountants, those customers least connected to our core product (one person described GitHub as the same thing as a pencil, he didn’t need to know what it was, but he needed to know who purchased it, so he could issue a check).
We investigated how organizations paid for GitHub (invoicing, annual billing, credit cards, etc.), and asked about how they paid for other software (Who is a great vendor?). We surfaced both the good and friction-filled workflows, and did a deep dive to learn how payments are processed and identified communication gaps that often result in late payments or accounting confusion.
By studying a payment process from multiple angles, we uncovered a splintered customer experience that was making it difficult for customers to purchase our product.
Pre-Release Programs
I also organize our pre-release programs to get early versions of our features out into the hands of people who partner with us to share feedback. Pre-release programs are largely qualitative and provide us with intimate insights into human workflows, tools, and social experiences that are not otherwise easily attainable with quantitative data. They help us to better understand how to refine the product and whether or not it’s something we’re ready to or should ship.
In this genre of research, sometimes we learn that we need to go back to the drawing board, because we haven’t gotten it quite right (yet!).
-
Git LFS – This was ground-breaking engineering work that make it possible to manage large binary assets with Git (Git is hostile towards large files), and a companion hosting solution on GitHub (our first paid add-on). We needed to identify the primary audience, use cases, and carefully introduce new technical challenges into our infrastructure (slowly, thoughtfully).
-
Direct Organization Membership – This was a large project that began with a team rewriting the underlying abilities potential and then designing new permissions (backend and frontend) to improve how people manage user membership and privilege inside of their organization, creating safer security models and leveraging changes to core features like team @ mentions most helpful towards collaboration.
-
Organization Authentication Policies – Companies like CircleCI, Gitter, Flowdock, and HuBoard run businesses built on GitHub’s API. They are security conscious and customer-focused. We conducted a two-part study to explore motivations and evaluate reactions to the design, documentation, and workflows associated with GitHub’s organization application policies.
Experiments
We run experiments to help product managers, designers, and engineers measure the impact of changes to existing features, the development and rollout of new features, and validate if an idea is worth pursuing. Our experiments typically go beyond clicks and are designed to better understand human behaviors.
Each experiment involves measurement of:
- Events – Short-term user actions with site UI, often with Google Analytics and internal event collection.
- Milestones – Longer-term accomplishments that tell us about behaviors and motivations.
Experiments are led by our analytics team, but when setting up the initial experiment and then later when writing an experiment report, qualitative research lends insights from the field to help explain why something is happening. You can read more about our experiments “Insights from Researching Growth”.