Tag: metrics

Goals for Upleveling Your Product Team

Goals for Upleveling Your Product Team

There’s a lot of content out there about metrics – OKRs, KPIs, AARRR, LTV, ARPU etc etc. There’s even some thoughts on this very blog about metrics (Metrics, Means and Maps, Cogitating on ROAS, It’s Good that Data is Man Made).

When product folks talk about measuring our team’s success, we always talk about the success of the things we build. Ideally the customer- or business-level outcomes that we want to achieve.

“This quarter we’ve prioritized reducing churn within the first seven days of a customer activation by at least ten percent. We can now observe that in six of our seven regions, churn has gone down by at least eight percent – we’re over halfway to our goal.”

There are still a lot of teams with the bad habit of measuring how MUCH they build – “We shipped fourteen new features this year!” – which is different than how something has impacted customers, the market, or the business. I know there are a lot of “Outcomes Over Outputs” folks, and to be clear, I’m not saying you shouldn’t ever measure output. Sometimes, in some organizations, it can be exactly what you need to do.

That’s a space to explore in a different post, I think.

It’s important – I’d even say mandatory, to measure and understand the success of the products that we build. What is missing in many cases is measuring the success of the team itself. If your product team were a product itself, how would you measure its success over time?

(Also worth mentioning, of course; teams don’t exist, but people do. )

When you’re leading a team, or building an organization, when thinking about the long term growth, you should be thinking seriously about how you’re investing in the success of the people on the team, and how to measure that over time, the same way you would for a new launch, a new product, or a new campaign.

Over a long enough time horizon, your team’s ability to accomplish those key business moving goals (decrease churn, grow customer base, improve LTV, move into a new market, etc) is downstream of this team development work. Building a strong team, and spending at least some time on how you keep that team improving over the long haul, is enormously high leverage.

Investing intentionally in the health, growth, and success of the professionals on your team also has an impact on your team’s retention, and ability to attract new talent.

These are the metrics that we use on my team – I’ll list them now, and then do a little commentary (as I do!). We’re sticking with these for 2024 (we do monthly in-team goal reviews and quarterly roundtables with our key partners) but there are some changes I think coming for next year. They’re sets of two metrics, in five themes or tenets.

2024 Team Development goals:

We Rise Together

  • Run monthly Product Jam sessions
  • Initiate 1x Development Opportunity per quarter (book club, speaker session, skillshare workshop, etc)

When I think about, “We Rise Together,” the “Together” is really broadly inclusive: it’s about creating opportunities for broad development and network growth within Disney. Disney has hundreds of product managers from cruises to in-park mobile apps to streaming players. Bringing folks together, to share skills and insights, and being visibly the team that is committed to doing so, generates a lot of positive visibility for the folks on the team, and makes new partnerships and internal collaboration opportunities much lower-friction.

It’s also quite fun! I personally get a lot out of a book club or tech talk; and it’s a nice forcing function to get me to actually read the new Cagan instead of scrolling Slack. Product Jam is a monthly 60-minute call where a member of the product org gives a talk or proposes a blog post or podcast to lead a discussion on. It’s quite casual but it’s a great chance to get to see folks around the org.

The tenet name is aspirational: we want to make our team better but not if it comes at the expense of our peers. It is also the first tenet on purpose: we rise but we rise together. Product is a team game, and supporting our colleagues and investing in the broader product org always bears positive dividends.

We See Far

  • Schedule 1:1s with PMs outside of Media Product 2x per Quarter
  • Schedule 1:1s with PMs outside of Disney Streaming 2x per Year

“Media Product” is the VP-level organization that we’re a part of. My direct lead is a Director, his direct lead is our VP. A PM outside of Media Product might work on the mobile app for Disney World, or the syndicator feeds for ABC, or the A/B testing framework for the streaming platform.

The important thing about this goal is that it helps us get out of our bubble: it’s very easy to get your nose so close to the grindstone that you lose sight of what else is happening around you. Taking some time to get to know folks outside of your direct team, as well as folks fully outside of our firm, helps to set a broader context about what’s happening in our company, as well as in the product field at large.

We See Far and We Rise Together are often contributive to one another: you have an intro 1:1 with someone, and they end up being an excellent Product Jam speaker. You get an intro to a PM working in an interesting startup from someone at book club.

We Care About Quality

  • Every L-XL+ project gets a Looker Dashboard & a Retro
  • Formally seek feedback from Engineering Teams Quarterly

This one is about great habits: we need to keep a pulse on how our projects perform, and how we might iterate on them in the future. To do this effectively, we have to intentionally build out the infrastructure that will fuel that awareness and iteration. The Looker piece just happens to be the tool we use – the tool is less important than getting in the habit of creating dashboards by default.

Any project sufficiently large to mandate a dashboard should also already have at least one retro scheduled by the engineering side of the house; but Product isn’t always invited to them. If it’s possible, I really like to sit in on those calls – they can often be a source of really helpful insight into process and approach, as well as feedback for how you personally are doing with a given team.

For the engineering feedback piece, we use a fairly simple Google Form, with a standard NPS metric and a few ethnographic and a few feedback questions. We schedule them in conversation with engineering leadership so we’re not asking engineers to fill out a survey in the midst of a stressful crunch time, or when lots of folks are on vacation.

There’s no perfect Product Manager, only the Product Manager that uses their product toolkit to best effect with the engineering teams they work with. Retros and quarterly surveys are a great source of insight as to how well your team is performing in that space.

Become Aware

  • Each PM attends at least 1 trade show in our industry per year
  • Each PM attends at least 1 Product focused event per year

This falls in much the same theme as above – a ‘trade show’ would be something like NAB in Las Vegas, a chance to connect with vendors, understand our space in the broader media and entertainment space, and walk around a gigantic trade show floor. A Product focused event would be something like Product Con or Lead Dev, maybe An Event Apart.

A theme-of-themes here is “get out of the office,” – mentally, network-wise, and literally physically. I am a firm believer that innovation emerges from novel intersections, unique combinations of experiences and expertise that are rare but valuable, and the most expedient way to achieve novel intersections is to get out there and experience a bunch of things.

Cause Awareness

  • Each PM speaks 1x per year at an external event or conference
  • Each PM speaks 1x per year at an internal event or conference

Becoming Aware and Causing Awareness are collaborative tenets as well – often we can check a couple boxes at a single event. I think for product folks, investing in public speaking, in clear communication, is so important, and is such a highly portable skill, even if folks I work with leave product forever (not because of me, I hope!), being clear communicators who can talk confidently in front of a room will serve them where-ever their journey takes them.

While the actual giving-a-talk is what’s being measured here, I feel that the preparation of a talk is in some ways at least as valuable as the experience of giving it. Much like blog posts, preparing a talk forces you to distill your thinking into something crisp, to take a position, and defend it to strangers.

Maybe at the end of the year I can do a roundup of how we’ve performed against these goals – and which ones we are shaking up vs. keeping as-is. It could be valuable to reflect on how setting them has been productive, or could be improved. Stay tuned!

Source & Medium: A Medium Sized Dilemma

Subtitle: Source, Medium, Attribution, Stale Information, and The Future of Data

Here’s our situation – we want to be able to slice reporting and dashboards by a number of dimensions, including source and medium.

MARDAT (the team I’m lucky enough to be working with) is working to make this kind of thing a simple exercise in curiosity and (dare I say) wonder. It’s really interesting to me, and has become more and more clear over the last year or so, how important enabling curiosity is. One of the great things that Google Analytics and other business intelligence tools can do is open the door to exploration and semi-indulgent curiosity fulfillment.

You can imagine, if you’re a somewhat non-technical member of a marketing or business development team, you’re really good at a lot of things. Your experience gives you a sense of intuition and interest in the information collected by and measured by your company’s tools.

If the only way you have access to that information is by placing a request, for another person to go do 30 minutes, two hours, three hours of work, that represents friction in the process, that represents some latency, and you’re going to find yourself disinclined to place that kind of request if you’re not fairly certain that there’s a win there – it pushes back on curiosity. It reduces your ability to access and leverage your expertise.

This is a bad thing!

That’s a little bit of a digression – let’s talk about Source and Medium. Source and Medium are defined pretty readily by most blogs and tools: these are buckets that we place our incoming traffic in. People who arrive at our websites, where ever they were right before they arrived at our websites, that’s Source and Medium.

We assign other things too – campaign name, keyword, all sorts of things. My dilemma here actually applies to the entire category of things we tag our customers with, but it’s quicker to just say, Source and Medium.

Broadly, Source is the origin (Google, another website, Twitter, and so forth) and Medium is the category (organic, referral, etc) – if this is all new to you I recommend taking a spin through this Quora thread for a little more context.

What I am struggling with, is this: for a site like WordPress.com, where folks may come and go many times before signing up, and they may enjoy our free product for a while before making a purchase, at what point do you say, “OK, THIS is the Source and Medium for this person!”

Put another way:  when you make a report, say, for all sales in May, and you say to the report, “Split up all sales by Source and Medium,” what do you want that split to tell you?

Here are some things it might tell you:

  • The source and medium for the very first page view we can attribute back to that customer, regardless of how long ago that page view was.
  • The source and medium for a view of a page we consider an entry page (landing pages, home page, etc), regardless of how long ago that page view was.
  • The source and medium for the very first page view, within a certain window of time (7 days, 30 days, 1 year)
  • The source and medium for the first entry page (landing page, homepage) within a certain window of time (7 days, 30 days, 1 year)
  • The source and medium for the visit that resulted in a signup, rather than the first ever visit.
  • The source and medium for the visit that resulted in a conversion, rather than the first ever visit.
  • The source and medium for an arrival based on some other criteria (first arrival of all time OR first arrival since being idle for 30 days, something like that)

It feels like at some point Source and Medium should go bad, right? If someone came to the site seven years ago, via Friendster or Plurk or something, signed up for a free site, and then came back last week via AdWords, we wouldn’t want to assign Friendster | Referral to that sale, right?

Maybe we have to create more dynamic Source/Medium assignation: have one for “First Arrival,” one for “Signup,” one for “Purchase.” Maybe even something like Source/Medium for “Return After 60+ Days Idle”

In the long run, it feels like having a sense of what sources are driving each of those behaviors more or less effectively would be helpful, and could help build insights – but I also feel a little crazy: does no one else have this problem with Source and Medium?

It’s Good that Data is Man Made

There’s a post from the folks at Highrise that’s been going around Customer Support and Success circles over the last couple of weeks: Data is Man Made, from Chris Gallo.

As someone who writes and speaks about customer support and leveraging data to do customer support better, I’ve had this article dropped to me in at least two Slack channels. Folks get a sense of mirth, I suspect, from needling me with articles and arguments that run contrary to the sorts of things I write about, and try to be persuasive around.

Yes; I will admit that I found this piece hard to swallow at first blush. Opening with…

Here’s a secret from the support team at Highrise. Customer support metrics make us feel icky.

… is a guaranteed burr in my side. Arguing against measurement from emotional premises?

Continue reading “It’s Good that Data is Man Made” →

Metrics, Means, and Maps

As a younger man, I spent a lot of time reading and discussing philosophy.

In the end, I was most attracted to modern moral theorists like Rawls and Nozick, but like all Philosophy majors at the State University of New York at Binghamton, I spent some time with all of the greats: Plato, Aristotle, Kant, Descartes, Marcuse, Arendt, and so forth.

(In fact, in the forward of Anarchy, State and Utopia, Nozick describes what I think is the most perfect description of all professional academia, not just Philosophy. I’m away from my copy, but I’ll post the passage when I get home!) edit: I gave it its own Post!

I’m bringing this up because one of my least favorite philosophers to read was Immanuel Kant. I struggled with Kant, like I suspect many 20 year olds do, as his writing is so incredibly dense, and translated from the original German. One piece of his moral philosophy that stands with me is this: to behave morally, a moral agent must treat other humans always as ends in themselves, and never as means.

To be more philosophically precise, Immanuel says never to treat other humans merely as means, but always as ends as well.  So, it’s not necessarily immoral to treat another human as a means, so long as you keep them in mind as an end also. It’s a tricky bit that’s easy to forget. Kant, he’s dense.

One thing that we need to bear in mind, whatever department we’re working in, is that our metrics are necessarily abstractions, a means to a larger end. In this way our mindset needs to be like Kant’s – some things are ends, some things are means, and we should be intentional about which is which, and remind ourselves that the distinction is important.

A quote that came up a number of times at the Growth Hackers convention this year was this: “Be careful what you optimize for,” and that, too, points at what I’m getting at here.

Our metrics, our measurable indicators of success, must necessarily be abstractions from real life. 

By this, I mean, reducing churn by 10% is only a means to a larger end, and has to be considered in that larger context. What’s the real reason? Why do you, personally and as an organization, want to reduce churn? Maybe it’s because you believe you have a product that can genuinely make peoples’ lives better, so the more folks who use it, longer, the better off they’ll be. That’s great! Maybe it’s to make more money – that’s OK too. 

In either of these cases, churn reduction is itself only a means toward a larger end. Success with this metric points to a larger success, something that you’re maybe not equipped to measure, something like Customer Happiness or Success of the Business. We need to keep this in mind.

Another quote that’s on my mind a lot these days: “The map is not the territory.”

Our metrics are only maps upon which we build our assumptions and beliefs – the underlying terrain, the real territory of your customers and your business, is far more complex, far more nuanced. Remember that we use metrics because they are abstractions, because they take our complex world that is impossible to understand all at once, and break it into easier-to-understand chunks.

Our metrics are by design not the whole truth. They’re reductive because they must be – because only by reducing a complex concept can we hope to make meaningful decisions. If our metric were the whole truth, if the map were a perfectly accurate representation of the whole territory, it would be perfectly useless.

Measuring our work, and our companies, and our success or lack of success, is absolutely vital to the success of any enterprise in 2016. Choosing the right metrics, and bearing in mind that our metrics only represent one part of the truth, is the hard part.