Tag: research

Don’t Taste One Coffee

Don’t Taste One Coffee

Did you know I had a whole career in specialty coffee before I became part of the tech industry?

It’s true – I trained hundreds of baristas, some of them competitors (barista competitions are a thing!), opened a number of cafes, developed a whole multi-course curriculum, I also kind of interestingly in retrospect did some product work with muffins?

One of the most memorable aspects of that time in my life was buying coffee, by which I mean, working with importers and exporters to source the green unroasted coffee to bring to the States, to then store, roast, sometimes blend, and serve in the cafes.

There are a whole library of ways that you can try to ensure taste (at a given price and availability) in the cup through the buying process. There’s a Sommelier type test for folks who do coffee buying professionally (which I did not pass by the way!) One of the most common tools in the toolbox is an exercise called cupping.

Cupping involves standing at a table, usually a very long table, where there are a collection of little groups of cups (the standard for professional buying sessions is 5, but I’ve been at tables with 3 or 4 cups per group, like in the image above), with each group holding a particular coffee from a particular farm. You might be tasting fifteen coffees, which would mean fifteen little groups around the edge of the table. Then, you and your colleagues and partners walk around the table, use your individual spoon to taste the different samples and different coffees, take lots of notes, and decide what fits your needs for the coming season.

Imagine instead, packing your bags, flying to Guatemala, taking a drive into the mountains, roasting, grinding, and setting up a single cup of a single coffee, giving that one lonesome sample a try, and your import partner turning to you and asking, “Well, should we buy it?

Even if you’d tasted hundreds of coffees in your life, it would be incredibly challenging to consistently make the right decision in that moment.

Humans are really good at comparing. When we have the chance to assess multiple options, in real time, all together, especially in discussion with others, we are much more successful when we have a small number of things to compare and contrast to one another, rather than having a single yes-no binary choice.

So we never only try one coffee. You try lots of coffees, and you keep trying them as they cool, to help get as much information about the different options, the trade-offs, and guide you in making the best choice you can.

The thing is, this is also true for making decisions in other fields, especially in Product, whether prioritizing a feature or making a whole roadmap, or hiring onto your team. You will always make better choices when you give yourself the space and process where you consider and discuss alternatives and the trade offs between them.

Something happens in the process of discussing trade-offs – it forces you (and your colleagues!) to get really clear on your thinking, and it can help draw out benefits and drawbacks that wouldn’t have arisen if you’d only tasted the One Coffee.

Teresa Torres (absolute luminary, her book is a must-read imo) outlines a great way to do this in the Opportunity Solution Tree exercise. If you give that approach a try, you’ll have lots of options to consider!

If we’re talking about the Ideal Platonic Product Organization (which doesn’t exist, by the way!), following our successful Opportunity Solution Tree exercise, we’d produce a ranked list of these potential opportunities to prioritize, and then we’d leverage our reliable, fast, and effective experimentation framework to run tiny inexpensive experiments to determine which possible direction would go the best with your customers and infrastructure, and then execute on that opportunity.

I can’t speak for other fields, but in developing software, Only One Coffee can sneak up on you.

“When can we schedule the Anti Light Particle Beam feature?” … when you hadn’t even heard about the Anti Light Particle Beam feature!

“Nautical Cloud Experience says we need to build the Zoidberg Platform v3 out by end of March to unblock their P Zeroes.” … but Zoidberg v2 was never even completed!

In situations like these, it’s incumbent on the Product organization to say, “Wait, hold on a minute, let’s think about what our alternatives are,” and then do so in a structured way. Remember, you always have the choice to do nothing! Between the proposed Single Coffee, and the evergeen Do Nothing, if you come up with only one or two other possible paths to a solution, you’ll already be in much, much better shape.

You won’t always get the chance to go through Discovery exercises, and you won’t always have the experimental infrastructure or organizational support to run lots of iterative experiments. But you can always ensure that you’re considering alternatives, and giving yourself and your firm the best chance at building the right thing.

The most important thing a Product organization does is avoid building The Wrong Thing. Never taste only one coffee.

Research in the Right Order: When to Interview Your Customers

One of the parts of my work that I get the most satisfaction from, and the part that most consistently surprises me, is in listening to our customers.

WordPress.com has a lot of customers. When you are dealing with a B2C company at this scale, it gets to be important not just to learn to listen, but learn to seek information in the right order.

(If you were at my SupConf talk, some of this is going to be very familiar!)

I like to talk to people – it’s part of who I am. I am an unapologetic talk-to-think-er. My most successful side hustle was a thinly-veiled attempt to get incredibly smart and incredibly busy cutting-edge farmers to talk to me. It worked! I interviewed them (and other members of their industry) for over a year. I think the art of the interview is a subtle one, and I’m the sort of person who literally reads books about different types of interviews.

Continue reading “Research in the Right Order: When to Interview Your Customers”

You’re Already Interviewing Your Customers

Let’s start with a story!

At Automattic, we’re lucky enough to have some pretty sophisticated internal tracking and analysis tools. I was recently involved in a conversation with my friend and colleague Martin, about a particular slice of our customer base, whose churn is higher than we would have expected.

One of the ingredients for this particular group of customers was that they had, at some point in the seven days before leaving our services, interacted with our Happiness Engineers via our live chat support offering. Given the tools at our disposal, we were able to pull together a list of all of these customers – and with the churn rate being what it was, and the total userbase for that product what it was, the list was not terrifically long. Double digits.

Some of you out there know this story, right? What better way to find out what is going on with your customers (or former customers) than asking them outright? Put together some post-churn interviews, offer an Amazon gift card, learn something new and helpful about your product or service. This is a pretty standard flow for researchers – start with Big Data to identify a focus spot, then focus in with more quantitative methods, interviews, surveys, what I think of as Small Data.

In this case, rather than jump to the usual move, and at Martin’s suggestion, I pulled up all of the chat transcripts, and read through them, categorizing them along obvious lines, pulling out noteworthy quotes and common understandings (and misunderstandings!) – treating these last live chats with churned customers like they were transcribed interviews, because in a real way, that’s what they are.

I was really surprised how insightful and interesting these live chat sessions were, especially when read back-to-back-to-back like that. In fact, I did not even feel the need to follow up with any of the customers, the picture was clear enough from what they’d already communicated with us. I was honestly floored by this, and left wondering: how much good stuff is already in these transcripts? 

Moving forward, I’m including customer email and live chat review as an integral part of any user cohort research that I do – it will allow me to come to the interviews three steps ahead, with far better questions in mind, and a much sharper understanding of what their experience might have been like.

Especially with robust data slicing tools, being able to cut down through verticals, cohorts and purchase levels means that I’ll be able to see a ton of useful, relevant conversations with customers similar to those I’m looking to learn more about.

This is also the case with you and your customers.

Even if you don’t have a user research team, or even one researcher, your support team is interviewing your customers every day. Even without data slicing tools, you can do something as simple as a full-text search on your last month of email interactions and get something close to what you’re looking to learn.

If you enjoy a support tool that has a taxonomy system or plugs into your existing verticals and cohorts, all the better.

This Small Data on your customers, these conversations, already exist. You don’t need to generate new information, you don’t need to sign up for third party user testing.

You’ve heard me say it before, folks – there’s value in the data you have. Use it!