Early GitHub: UXR Projects
2015-05-10
In 2013, I joined GitHub as their first user experience researcher hired to grow our research practice. To date, all of our research projects have been conducted in partnership with teams of product managers, engineers, designers, and writers). You can't give empathy away.
Here is a high-level roundup of a few of my favorite projects from my past two+ years at GitHub:
Foundational Research
Long-running research studies with many actionable insights falling out along the way:
Understanding project management
Shipping software can be a simple to complex project with all the challenges you can imagine on a technical level with tooling and a social level with communication and process. A lot happens in between both. We started our foundational research with a survey and quantitative analysis of interactions on the site. Still, it can be tough to interpret quantitative data if you haven't been in the field looking at what's going on.
Part I. Survey: The guiding purpose of the survey project was to help us understand how GitHub users manage software projects so that we can develop features to meet their needs better. We designed a survey that asked about people's experiences with workflows, tools (including those external to GitHub), and team size. We received responses from about 2,000 randomly selected logged-in/out users.
The survey data turned out to be so rich that we presented results in three reports:
Process report – Described survey methodology and sample, what we learned about the workflows and methods people use to manage software projects, and identified sources of stress and friction.
Tools report – Surfaced insights into users’ experiences using GitHub Issues to manage projects and competitor products like JIRA, Trello, Pivotal, and Asana. The survey asked users about the features they most wanted in a project management tool and the factors that keep them from using GitHub Issues.
Opportunities report– Tied together insights from the Process and Tools reports and examined the blockers that prevent GitHub users from relying on Issues for project management. We did a deep dive into the open text to investigate users’ needs and priorities in their project management systems and identify where GitHub has the most significant opportunities to build on and improve workflows.
Part II. Interviews: Qualitative research seeks out the things that are hard to measure, and watching and listening to people can give you insights and help inform and explain patterns from existing quantitative data and future quantitative studies.
I sourced candidates for team-attended interviews from our most interesting survey respondents (part 1.). The results of the interviews provided us with greater detail than the quantiative data alone and helped the team decide on the product strategy to pursue.
New user experience on GitHub.com
People who signed up for a GitHub account in 2008 are different than the people signing up for accounts today. To help our newest users achieve success, we conducted a foundational study to:
Learn about the mental models and experiences people bring concerning version control.
Discover why people are signing up (go deeper than traffic patterns).
Surface data about what perceptions people bring to the application (what do people want to do?).
Ask how we can help people the most (e.g., content recommendations, tutorials, etc.).
To start, members of our product team attended interviews with a diverse set of our newest new users and people with more than two years of tenure. Our interviews were designed to help us understand how people learned and taught git and Github. We asked questions to understand motivations better, walked through workflows, and listened to participants excitedly discuss their aspirations.
The outcomes helped us develop a new user survey, presented after sign-up and before the first run. The survey data helped us create a set of predictors useful for understanding baseline criteria for success and failure, which we then used to inform a set of experiments (noted below) and an exit survey to identify why people go quiet within their first 30 days.
Product billing and payment
To sell a product, you need to have a clear understanding of purchasing flows and experiences. I worked with the engineers tasked with refactoring and improving GitHub Enterprise's purchasing flows (account management, invoicing, upgrading, and self-serve experiences).
This study challenged me in many ways, namely because it required recruiting and interviewing accountants, those customers least connected to our core product (one person described GitHub as the same thing as a pencil; he didn't need to know what it was, but he needed to know who purchased it so that he could issue a check).
We investigated how organizations paid for GitHub (invoicing, annual billing, credit cards, etc.) and asked how they paid for other software (Who is a great vendor?). We surfaced both the good and friction-filled workflows, did a deep dive to learn how payments are processed, and identified communication gaps that often result in late payments or accounting confusion.
By studying a payment process from multiple angles, we uncovered a splintered customer experience that was making it difficult for customers to purchase our product.
Pre-Release Programs
I also organize our pre-release programs to get early versions of our features into the hands of people who partner with us to share feedback. Pre-release programs are mainly qualitative and provide intimate insights into human workflows, tools, and social experiences that are not easily attainable with quantitative data. They help us better understand how to refine the product and whether or not it's something we're ready to (or should) ship.
In this research genre, we sometimes learn that we need to return to the drawing board because we haven't gotten it quite right (yet!).
Git LFS – This ground-breaking engineering work enabled managing large binary assets with Git (Git is hostile towards large files) and a companion hosting solution on GitHub (our first paid add-on). We needed to identify the primary audience use cases and introduce new technical challenges into our infrastructure (slowly, thoughtfully).
Direct Organization Membership – This was a large project that began with a team rewriting the underlying abilities potential and then designing new permissions (backend and frontend) to improve how people manage user membership and privilege inside of their organization, creating safer security models and leveraging changes to core features like team @ mentions most helpful towards collaboration.
Organization Authentication Policies – Companies like CircleCI, Gitter, Flowdock, and HuBoard run businesses built on GitHub’s API. They are security-conscious and customer-focused. We conducted a two-part study to explore motivations and evaluate reactions to the design, documentation, and workflows associated with GitHub’s organization application policies.
Experiments
We run experiments to help product managers, designers, and engineers measure the impact of changes to existing features, develop and roll out new features, and validate if an idea is worth pursuing. Our experiments typically go beyond clicks and are designed to understand human behaviors better.
Each experiment involves the measurement of the following:
Events – Short-term user actions with site UI, often with Google Analytics and internal event collection.
Milestones – Longer-term accomplishments that tell us about behaviors and motivations.
Our analytics team leads experiments, but when setting up the initial experiment and then later when writing an experiment report, qualitative research lends insights from the field to help explain why something is happening. You can read more about our experiments, "Insights from Researching Growth.”
