My last day at GitHub also marks my third anniversary founding and leading GitHub’s research team. I’ve had the incredible opportunity to work with brilliant individuals on a great product.
As much as I am proud of the projects I worked on, I am honored to have worked with outstanding human beings. The company has changed. My heart breaks to be leaving, but it is also full at the same time.
Here are some recent projects of which I’m most proud:
1.) Octostudy 37: Protected Branches / Required Status Checks
This qualitative study was designed as a pre-release with 20 GitHub.com organizations. I divided the organizations into two groups:
- Group A. received the feature passively with documentation.
- Group B. received the feature with intervention in a think aloud.
The product team learned numerous things about confusing UI as well as blockers and needs for GitHub.com administrator users, which led to much more successful ship, better foundation for documentation, and reduced support needs.
2.) Octostudy 38: Tools & Workflows, “Transformers & Ecosystem”
This project was designed as a comprehensive study of the tools, workflows, and background of developers on GitHub.
The survey instrument was 35 questions long and its thoughtful design allowed us to look at data from several different perspectives for different purposes, helping numerous product teams (reports included: Transformers & Ecosystem, Journeys, GitHub-in-Three, New Account Creators “NAC”, and GitHub Demographics).
3.) Octostudy 39: GitHub Organizations Improved User Permissions
This 12-month project to design and implement a new user permissions system rolled out in three phases with 20 organizations and culminated in a complex controlled experiment and slow release to GitHub’s general population.
4.) Octostudy 40: GitHub-in-Three
As part of the annual GitHub Tools & Workflows survey we asked an optional open-ended question at the end:
How would you describe GitHub in three words?
Respondents answered using a single open-text field, which was intentional as part of the instrument design. However, depending on their interpretation of the question, some people wrote three-word sentences, while others listed out words, and a few went beyond the constraint of three.
This generated a lot of messy data, which allowed us to try several types of analyses. In the end, we didn’t have enough data to constitute “big data,” but we also had too much for two researchers to go through alone. The resulting report was stunning and we were surprised at how few times the Octocat was mentioned, which impacted changes to how we were considering evolving our brand strategy.
5. Octostudy 41: GitHub Journeys
This report is a companion to the Tools & Workflows: Transformers & Ecosystem study. In that study, we explored the toolsets employed by long-term GitHub users, surfacing challenges they faced in becoming fluent with Git, and how they solved or didn’t solve those problems. Here, we broaden our scope to include a large sample of new account creators, with two aims:
1.) Gain insights into how developers build their toolkits, observing how toolkits change or stay the same over time based on behaviors, motivation, and tool and service availability.
2.) Surface similarities and/or differences between GitHub’s core users (people with an average tenure of three years) and new account creators (~90 days).
Pairing data from these two groups provides us with the freshest perspective we have on who is using GitHub today. This report should help product and marketing teams identify user behaviors we want to nurture as well as those we might want to change (e.g. moving from Sublime to Atom) for both tenured users and new account creators.
6. Octostudy 42: GitHub 365
This project was designed to fill a blindspot that surfaced from the New Account Creators study– tenured inactives: those with accounts six months to one year old, who had been inactive for at least 3 months.
We learned that more than half the accounts we consider inactive, consider themselves temporarily dormant. This information is crucial to our understanding of what we have historically considered “inactive,” as people consider GitHub for project-based work and expect times of dormancy.
Summary
Among my most important contributions to GitHub Research is that in 2015, I had established a cadence of repeatable studies, which make it possible for us to measure change over time, including:
- Bi-annual GitHub Enterprise NPS & Product survey (high-stakes customers)
- Atom foundational survey
- Annual “Developer Tools & Workflows” survey
- The NAC Longitudinal study (new users)
- Annual “GitHub 365” survey (inactive/dormant users)
Each survey included an option to opt into GitHub Research opportunities like usability testing, early feature access, as a results we have more than 10,000 volunteers to use for recruiting efforts.
PostScript: Thank you Kyle for the opportunity to join GitHub. You were an amazing colleague and friend who took a chance on me and changed my life.