Photo credit: Lee Chatfield
“However small the chance might be of striking lucky, the chance was there.”
― Roald Dahl, Charlie and the Chocolate Factory
This is the story of an experiment where we gave our hosted product (GitHub.com) away for free. What happened truly surprised us!
To begin, GitHub.com’s business model charges for private repositories with two types of paid plans:
- Personal – Structured for personal private workflows. Collaborators may be added. Does not offer “team features.”
- Organization – Structured for team workflows with features like team management and team @ mentions.
What follows recounts some of the high-level insights we drew from this experiment that can help inform your research strategy for better understanding a product’s value, price, or both.
We used a classic experimental design with 40k tenured users on free accounts assigned randomly to treatment and control groups. The treatment group was divided randomly into three subgroups:
- 6,600 people — received a coupon for 1 free repository
- 6,600 people — received a coupon for 3 free repositories
- 6,600 people – received a coupon for 5 free repositories
In addition to measuring activity, people redeeming the coupon and creating private repositories), there were two other data-gathering components to the study:
Screener data, which collected from coupon redeemers (4,418 participants).
Exit survey data, which was sent to all members of the treatment group (2,039 respondents).
We anticipated chatter on Twitter and Hacker News, so we rolled the experiment out in small batches, keeping our eyes on the public noise.
The experiment involved 15 days of subject enrollment and three layers of data to measure (screener, user activity, and exit survey data), which made reassembling the complex data a bit maddening, while also interesting and fun.
Understanding attitudinal data
Our research team set out to measure three things:
- Coupon redemption
- Repository creation
- Perception of value
Both coupon redemption and repository creation are trackable behaviors with data that comes directly from the GitHub.com database, including who: clicked on our link, submitted the screener, created a new repository or migrated an existing one, and various other discrete activities.
“Perception of value” or as we like to call it attitudinal data, and comes from the exit survey. We designed the exit survey instrument to shine a light onto why people did or didn’t redeem their coupon and/or did or didn’t create their repository. By the time we are analyzing attitudinal data, we also had more information on respondent attributes (member of a free/paid organization, geography, OS, etc.).
Attitudinal data gives us greater insight into what levers we can pull on with marketing and product experiences to effect change in behaviors like coupon redemption and repository creation.
To our surprise, just over 4,000 of the possible 40,000 people redeemed their Golden Ticket and far fewer went the extra step to create repositories after redeeming their coupon. We learned that people who were members of an organization on GitHub or who received only one free repository were more likely than their peers to take advantage of the offer. These insights told us something important about how responsive different user types are to our outreach efforts, and how they use and value our product.
Value. Organization membership likely impacts redemption rates through attentiveness to notifications and awareness of GitHub’s value.
Attention/Engagement. Members of organizations, particularly paid organizations, have relationships to communities that demand frequent attention and engagement. These members are likely to be more attentive to notifications from GitHub, which normally indicate project activity.
Project work. Members of organizations may also be more likely to perceive value in private repositories, and have projects to work on that could benefit from a private space.
Our ability to influence user beliefs is conditioned by brand trust and loyalty, users’ own propensity for skepticism, and their use patterns- how they use Github, for what purposes, and who they use it with. Those are things we can’t measure with instrumentation in the GitHub application, so we have to rely upon self-reported data from surveys and customer interviews.
Mixed Methods Insights
At first, the quantitative data alone told a simple and disappointing story. Not many people redeemed the offer and far fewer created repositories.
However, the role of a researcher is to sift through many quantitative and qualitative insights to find the story. We used answers from the category of “other” and the final open-text response option to help guide our analysis.
Specifically, the very last question on our exit survey asked:
If we could grant you one wish to improve your experience with GitHub, what would you wish for?
at least five private repositories, or free private repos for teams up to five people
Free private repos for small teams, a la Bitbucket.
Private Repositories, for teams, for free
We realized that people were reading “free private repositories” as “private only to you” vs. “private to you and friends you invite to work on a code base.” (When we realized this, I personally smacked my forehead!) Would more people have redeemed their Golden Tickets if they knew they could invite collaborators to their private repositories?
Placing a value on GitHub Goods
People tell us that they want lots of things when they’re free, especially repositories. Asking respondents in this sample to choose between many valued things provides information on the ordering or prioritization of these preferences.
We asked both people who had redeemed but not used and those who hadn’t redeemed their free repositories, to share which of the most frequently requested goods would be most valuable. Response options provided are deliberately disparate; not just things that we can or would actually provide (e.g. merged PR), but things that people find valuable on and around GitHub. Surprisingly, there’s not that much difference in these answers between the segments that redeemed their coupon and those who did not.
Our favorite open text response for the category of “Other” was easily: Github underpants.
If you were studying value, what would you put on your product’s list?
Importantly, when we’re dealing with money we’re also dealing with humans. We confirmed that people don’t do things just because you send them a “free” product offer.
Humans do things because you send them something of value and they believe that using the thing you sent that will cause them to change their behavior will also benefit their lives. We additionally learned that delivering something by email, especially with the words “free” and “private” might not have gotten us off to a great start (note: in spite of sounding like salacious spam we verified that our deliverability was quite high).
Feature awareness of the campaign and how to use GitHub is strikingly low. Many participants shared that they wished for “tutorials” to learn to use the free private repositories, indicating that “instructions are needed.” We also received feedback that summer vacations may have been a factor for why people missed the email or did not participate in the campaign.
“Private repositories” may be an ambiguous description. A number of participants stated that they wish GitHub would provide private repositories for “teams,” so they could work with other people. We suspect that adding collaborators to private repositories is too opaque, and that “private” suggests working alone.
One free repository feels special. While the number of repositories we offered (1, 3, or 5) to users did not affect redemption behavior, it did have a significant effect on use of the repositories. Those receiving just one free repository were more likely to use it than those who received three or five (25% versus 16%). However, in open-text comments we received feedback that one wasn’t enough for people who develop with modules, freelancers who have multiple client projects, and people asked for repositories that could be used by small teams.