Measuring Hard-to-Measure Things

Posted by Chrissie Brodigan on June 23, 2015

What? is not the same thing as Why? You need both, but why is usually two steps ahead of What?.

Last week I spoke at Monitorama, and it was one of the best experiences of my career. The community was accepting, supportive, and safe. Thank you for the warmest welcome, truly. Jason Dixon, you are the best! The following post provides a bit of narrative to accompany slides and video.

Slippery Slope

I joined GitHub in March 2013, excited and nervous to be starting as the company’s first user experience researcher. Professionally, I’m trained as a historian who turned towards design to fund an academic habit. Not being an engineer researching engineers felt intimidating, but it turns out there’s plenty of overlap between disciplines like: curiosity, skepticism, and a compulsion for discovery.

The first thing I did was research our internal project history. What did my new team members think about research? That endeavor surfaced a freshly-pressed debate about variance testing, usability studies, and user research, inspired by a thread on Hacker News. One comment, in particular, stood out:

We make informed design decisions by using the product we build on a daily basis. We keep what works, remove what doesn’t and improve what needs to be improved. I think formalizing that process and having a “UX Researcher” is a slippery slope.

Since I’d been hired shortly after this post, it was clear that we were on a slippery slope. However, my colleague’s insights were both honest and accurate, and GitHub had been wildly successful without user research.

In its early days, GitHub relied heavily on the experience and intuition of its many talented engineers and designers. The creative combination of people, vision, ability, and hunger to build a product that solved their own problems with version control, code hosting, and collaboration was a huge hit. Developers, especially those from the Rails community, shared similar needs and enthusiastically signed up and started using and paying for GitHub.

Powered-by Luck …

Without research all you have is luck. –@sboak

Early product success is exciting and terrifying, because if the growth path is up-and-to-the-right your audience will broaden as will your challenges. Newcomers will arrive with experiences less like your earliest users and less like your own, and may begin using your product in ways you didn’t intend. This may leave you puzzled, scratching your head.

Who are these people? What do they want to achieve? Why do their goals matter? How do we help them?

In GitHub’s case, there was a distinct shift in growth and audience. By 2013 developers experienced with version control and git signing up for accounts were far outnumbered by newcomers with limited experience, hungery and expecting to learn both. The caché of a GitHub profile and membership to its community brought many types of people through the door and into a product that is frankly pretty hard to figure out.

I asked participants in in our first foundational study of new user experiences to draw GitHub’s core features. This drawing is still one of my favorites, she used colors so deliberately and keenly illustrated confusion between branching and forking:


Why would you request someone “pull” in your changes? What would a person think if you tried to change their project?

As your audience broadens early luck can start to run out.

Blind Spots

All products have blind spots, and no amount of analytics and monitoring can account for the hidden variables “waiting to screw you.” Even the best systems can gather data on unusual behaviors for a long time without signaling that something is amiss. The sheer volume of data we have now can make it hard for talented analysts to utilize information, and powerful tools and visualizations are rendered less useful.

Gap from metric/graph to insight can be huge. –@randommood

Graphs are impressive and addictive. They depict what is happening (e.g. year-over-year growth, anomaly detection, etc.). However, in isolation, graphs can also be misleading and even create a sense of overconfidence in data. No amount of gorgeous graphing or intricate analysis can make up for fundamentally flawed data.

It’s hard to have faith in the results if you don’t really know why something is happening.

What v. Why

What? is not the same thing as Why? You need both.

Why is hard. Why requires you step outside of your comfortable universe and uncomfortably step into experiences unlike your own. Why is time-consuming and awkward. Why can’t be automated. Why requires you look at the world through someone else’s eyes. Why breaks our product design egos down with radical amounts of empathy. Why shows us what we don’t know and what we didn’t want to see. Why confounds and humbles us and is usually two-steps ahead of what. Why can sink a product or push it over the mountain into success.

It’s hard to measure “Why?” without sitting down to watch someone use your product. The experience of interviewing someone or sitting with them as they navigate through tasks provides intimate insights into human workflows, tools, and social experiences that are not otherwise easily attainable.

Watching someone use your product will change the way you see your product.

Human Instrument

With regards to engineering challenges, data collection, and analysis we tend to think in terms of technical instruments. If you want to surface blind spots, you will have to get out from behind the numbers and in front of people. Think of yourself as a human instrument. You are handcrafted, possibly artisanal, and definitely brilliantly designed to measure socially meaningful questions with an open and compassionate heart.

Humans do really interesting things with software when they’re confused and frustrated. They can become very skilled at designing effective workarounds to mitigate bad design, and unwinding those patterns can be difficult.

When you talk with people who use the product you’re building consider these three things:

  1. Goals – People “hire” your product to help them achieve personal efforts and ambitions.

  2. Motivations – People are driven towards achievement, they have underlying reasons for behaving in particular ways. People want to succeed.

  3. Workarounds – When goals and motivations are out of sync with a product’s user experience, people can be very effective at using bad design.

Change, even good change, can be frustrating. Adapting to change requires a person make an investment of trust and time adopting new behaviors on the way towards achieving aspirational ones.

*Read more about designing for core, new, and aspirational behaviors and learn from some of the mistakes we made with a big project.

Vocal Minority

When you’re designing a product, especially one that needs to scale for thousands and millions of people, you must be increasingly deliberate in your design decisions. If you introduce complexity, you must also consider its effects on both current and future behaviors; it’s hard to take features away.

There is always a vocal minority.

Listening closely and reacting to insights from the wrong audience might encourage you to head in the wrong direction. Sometimes the vocal minority are not the best representation of your customers, they just happen to be the ones generating the most feedback and are often easier to reach.

Don’t take the easy path because it’s easy. Measuring hard-to-measure things is hard work, and it’s hella fun too.


At a time when we are awash in data, if you think in terms of human behaviors and get out from behind the numbers to talk with people, I guarantee you will become better at discovering, measuring, and solving hard problems.

*Related: Measuring Hard-to-Measure Things and video