Category: General

# Crowd Sourcing Organizational Improvement

### Subtitle: It’s Called the Analytics Road Show!

Here’s the situation: our company has a data organization – it’s probably kind of like your company’s data organization: it has some data engineers, it has some governance experts, it has some analysts, some developers.

We’ve been making great strides in doing the right work, and getting better at delivering that work quickly, accurately, and in communication and consultation with our stakeholders.

But, that feels like table stakes, right? One lesson that really rings out to me from my time before I worked in tech, a lesson from one of the owners of the chain of bakeries where I taught over one hundred baristas how to do latte art:

You can’t work on the business if you’re always working in the business.

(This idea I believe originates with the book The E Myth ? Correct me if I’m wrong on that one though!)

This is something I’ve been cogitating on a lot these days: not just, how do we do what we do, and do it well? But, how do we improve the improvement? How do we improve our processes, our structure, the whole way we think about and engage with our data, with our measurements – even how we engage with one another within the organization.

So – I think I need to get outside of the organization to get greater insights here: I’m taking this show on the road. I’m calling it the Analytics Road Show. I have a deep and abiding love of chatting with folks – some might uncharitably call it nosiness – which I am hoping to leverage into a bunch of sit-down sessions with folks working in similar organizations but not mine.

Getting outside the building is a key part of this endeavor: I need to get at this with a beginner’s mind. So then, dear readers, where can I find folks willing to talk with ol’ SAO?

I have the great fortune of being a member of the Locally Optimistic Slack community (you should join us!), and when I dropped this into the #nyc channel:

… I got a serious no-joke resounding response. So, here goes nothing! July 15th and 16th (that’s next Monday and Tuesday!) I’ll be heading down the mighty Hudson to have coffees, lunches, and mid-level IPAs with some brand new friends in NYC.

I am really looking forward to this, as well as recording my thoughts in standard blog-post format for y’all – and internal action plans for my colleagues.

# The Pitch for Looker at Automattic

Automattic, and the part of Automattic where I work, WordPress.com, has a lot of data.

A really tremendous amount of information: so much information it’s passed the point of being helpful, and has started to become a hindrance.

Where do you find the information you’re looking for? How do you know which potential source of information is correct? Do you want the numbers that accounting uses? The numbers that the billing system uses? Maybe a combination of the two – but wait, shouldn’t those numbers be the same?

One of the big goals, part of the vision of my current team, Marketing Data, is to figure out how we can be better ambassadors of our information to our colleagues. How we can serve as translators, or maybe sherpas (sherpii?), on this journey we’re all on together.

The importance of this goal is kind of abstract: we hire a lot of folks who are really brilliant, who shine in their areas of expertise and are poised to create explosive value for their team, their division, and the company at large.

For many of these folks, their background isn’t particularly technical: they may be super Excel savvy (which is, I believe, equivalent to programming in many ways!), but when it comes to directly querying and manipulating raw data with SQL or other querying languages, it’s too much – and I am sympathetic to that.

The way I see it is, we’re putting two obstacles between these folks and their ability to realize their own greatness – and to help maximize the velocity of their impact, how quickly they’re able to go from curiosity to insight to revenue change.

The first obstacle is technical: it doesn’t make sense for them to learn the ins and outs of our data as well as becoming fluent in querying languages. So, they have to make requests of data professionals or software engineers to get the information in their hands that will allow them to maximize their value – and in some sense, their own personal growth.

The second obstacle is social: as a good colleague, no one wants to feel like they’re wasting the time of the folks they work with. But this is the way that we’re making our comrades feel – maybe not intentionally, but implicitly. If, as my colleague Demet often says, only one out of any ten A/B tests will produce results, and those tests have to be analyzed by hand, the nine without results will have the added result of the analysis-requester feeling a bit sheepish.

When there exists a piece of friction between curiosity and insight, when a professional has to ask a question of an analyst or engineer before they can validate their curiosity and pursue insight, we attach a tiny (sometimes not so tiny) cost onto that question. Great marketers ask tons of questions, because they know that it’s only through curiosity and exploration that they can find the kinds of insights they need to discover explosive growth.

When that friction exists because they can’t explore their curiosity directly, but only through other folks, we’re doing them a disservice, and reducing their ability to do their work.

Therefore, any good pursuit of that overall goal – being good ambassadors or translators for our data to less-technical members of our professional community – will work to break down those two obstacles.

Enter Looker.

We tried a few different Business Intelligence tools, and there were a few contenders for our attention, but I kept hearing about the potential of Looker, and especially notably, the importance of its modeling layer.

It took me a shamefully long time to understand what this thing is – I here humbly tip my hat to Matt Mazur and Stephen Levin and the rest of the good folks in the #Measure Slack channel for being so patient and generous with their explanations – and it turns out they were right, the particular features of the modeling layer really are a game changer.

Here’s why:

Imagine that you’re onboarding a new member of your data team: part of that onboarding will be cultural and social but a big part of it will be about relationships between different pieces of data – probably mostly tables but maybe also different sources. You’d say things like:

This table contains every user id, hopefully only once, along with some facts about each customer.

This one is every transaction, with a receipt id, and a total paid amount. The user id is in here too, but many times, since a single customer can have many transactions. If we want to join this table to that table, we need to remember that many-to-one type relationship, or we’ll have problems.

… and so on. In a sense you’re trying to take all of your own understanding about your data, and the way that it all clicks together to form a cohesive whole, and communicate that understanding into someone else’s brain.

In a Matrix style future, you’d be able to plug a plug directly into your new hire and sort of transmit that understanding directly into their head, so they could behave as though they had all of your hard-won knowledge – along with everyone else who made use of that data, because, why not?

Well, that’s sort of the modeling layer.

The modeling layer is a way of defining, through code, all of the relationships and nuances of your existing data buildout, and then presenting them to folks in a useful way, with those relationships in place. They can ask questions and explore the data as though they had the sum total understanding of everyone who built the modeling layer.

In some sense you already have a modeling layer – it’s just in your head, and can only be shared by explaining it to other folks. What Looker does is it gives everyone on your team – everyone in your company – the same powers as Rogue from X-Men.

The modeling layer can literally act like a superpower. Folks don’t have to understand how the data is stored, how the tables relate to one another, they don’t have to wake up in a cold sweat with the date/time doc from Apache Hive seared into their subconcious (Is month `MM` or `mm`??) – they just have to use a nice, clean GUI with solid built-in viz. And, they don’t have to ask anyone for help.

Looker gets us past our two big obstacles – once your data is modeled (which is not easy or fast but it is worth it), there is no longer a technical requirement for folks to explore the data, and there is, in the majority of cases, no social obstacle either.

Thus far it has been the right choice for us, and it’s something I look forward to working with for a long time.

# SAObattical: Return and Reflection

This post is my reflection on my sabbatical and also lots of pictures. I’m adding a Read More so it doesn’t clutter up the feed for folks who aren’t interested.

The most important thing is that I am grateful for a company with such a benefit, grateful that I get to work here, and thankful for a team that continued to succeed in my absence.

# Cogitating on Return on Ad Spend – AKA ROAS

I’m still pretty new to this whole marketing thing: I’ve been a part of Automattic’s marketing efforts for just over a year, and I feel like I’m still learning: the pace of education hasn’t slowed down even a bit.

One of the things that was a real challenge for me was getting to understand the language of the work, especially given our interactions with a number of outside vendors and agencies: the number of acronyms, shorthand and unusual usage of otherwise common words is a huge part of the advertising world, and it serves many purposes.

The import of accessible language is probably something I should save for its own post: I think that, especially in highly interdependent company like Automattic, opaque language, complex jargon, and inscrutable acronyms are more of a hindrance than a help, and in fact likely do us harm, given the way that we, as humans, myself included, want to feel smart, and powerful, and it can be very attractive to nod along rather than ask hard questions.

If you’ve been following this blog for a little while, you know that measurement and the implications of measurement are things that I think about – here’s a piece about metrics generally.

My broad position on metrics is, they’re reductive, necessarily and usefully so, and need to be understood as means rather than as ends.

All that to say, we should also be careful not to treat our metrics as being perhaps more reductive than they really are, or to behave as though what we are measuring is simple, when in fact it is not simple at all.

Taking something complex and making it simple enough to be useful – that’s the essential core of all measurement. Taking something complex and acting like it is something simple is another thing entirely, and a very easy way to increase your overall Lifetime Error Rate.

This brings us to Return on Ad Spend, sometimes shortened to ROAS. Return on Ad Spend can be calculated like this:

$\dpi{150} \fn_jvn \LARGE \frac{r}{S}$

…with being revenue and being spend. Generally the output is represented either by a ratio like 3:1, where for every dollar you spend on advertising, you get three dollars worth of revenue, or with a percentage – 3:1 would be represented as 300%.

It looks pretty simple. It’s generally referred to as being very simple, or easy, that kind of thing. Which, well, it is, at least on the face of it.

(The rest of this Post is about the sometimes hidden complexities of ROAS. If you want to learn more about using the metric in a tactical way, John at Ignite Visibility has a great write up on how to calculate and break out ROAS, as well as some wrinkles about attribution, which I recommend if that’s what you’re looking for. Here’s a link)

Let’s talk about this metric: ROAS. The name holds a lot of promise, right? Return on Ad Spend: something everyone who spends money on ads wants to learn, the dream of marketers everywhere. How much are we taking back in, for the amount we are putting out?

The trick of ROAS is, we have built in a set of assumptions: specifically, that the numbers we put in represent the whole of each of those categories. The trouble here is that there are only very specific parts of the marketing spend where that is a safe assumption: low-funnel type tactics, especially for e-commerce companies shipping physical products.

In these situations, for these companies, ROAS tends to be a clean metric: you have a very clear picture of where you are spending money, and each transaction has a straightforward, static revenue.

The trick is,

For SaaS companies, ROAS can become much more complicated: imagine your company sells a single product, some type of Helpful Business Software, and it retails for \$100 / year. If you run some numbers, you find that you spend on average \$50 in ads to get a customer – this looks good, right? We can say we have 200% ROAS and call it a day.

Of course, one of the great advantages of having Data is that we are able to record it, and then see how it changes over time, and try to do the sorts of things in our business that move the needle in our desired direction.

For a SaaS company, two of the metrics that you live or die by are Customer Lifetime Value (sometimes called CLV or LTV) and the dreaded Churn Rate – astute readers will note that these two metrics are inextricably linked. Briefly: LTV is the amount of revenue that your business can expect to make from a given customer, and the dreaded Churn Rate is the expected number of customers (generally at a rate out of 100, represented as a percent, like: “Our Dreaded Churn Rate is a spooky 13%!” )

A saavy SaaS marketing analyst will use the expected lifetime value of a customer in the top of the fraction up there, to determine Return on Ad Spend – for two great reasons. FIRST, because it is more accurate: if you’re looking to determine the total return it makes more sense to use LTV than simply the ticket price. SECOND, because it will make her look better in her reporting.

Consider: for this same sale of our Helpful Business Software, our expected LTV isn’t \$100, which is the annual cost of our product, but rather, \$200. This doubles our ROAS. This is great news!

(It’s not really news at all though, right? We’re not actually improving either our ads or our product, we just used a more accurate number. Metrics are means!)

One wrinkle, though, is that now we’re not really using that equation above anymore – we’re using something more like:

$\dpi{200} \frac{LTV}{S}$

If you’ve ever spent any time trying to calculate your customers’ lifetime value, you know that this has suddenly become a much more complicated metric.

What happens once we start to bring in more complicated ingredients into our ROAS pie here, things like LTV, is that ROAS moves from being a static sort of snapshot into a metric that is much more dependent on other parts of the business to be successful.

In the above example, imagine if your company has had a disastrous year, and your Dreaded Churn Rate has skyrocketed, driving your LTV down to below \$100 (due to let’s say sweeping customer refunds and growing customer support costs) – now our ROAS is below 100%, even though literally nothing has changed on the advertising side. In this situation, ROAS becomes a larger aggregate metric, telling us something about the business at large.

This brings to mind a larger question: do we want ROAS to be a heartbeat metric, an indicator of the business overall? Or do we want it to be what it was about a thousand words ago, a simple snapshot of how our advertising efforts are going?

As we move away from direct retail e-commerce businesses into more complex companies, and up what’s called the advertising funnel, ROAS becomes additionally tricky, not because the equation itself becomes more complicated, but because we start to introduce uncertainty, and even worse than that, we introduce unequal uncertainty.

Generally, you know how much you’ve spent. This is true even for less measurable marketing efforts, things like event sponsorships, branding, and so forth. What you decide to include is a little bit of a wrinkle: do you include agency fees? Payroll?

The uncertainty comes into play in the revenue piece, and this is why ROAS as a metric starts to break down as we move up the funnel, because the lower part of your fraction, your spend, stays certain, while the upper part, the revenue, becomes increasingly uncertain, which makes the output more and more difficult to use in a reliable way.

This is a problem that crops up a lot in marketing metrics, and something I’ve been thinking on quite a lot: we often will compare or do arithmetic on numbers which have wildly different underlying levels of base uncertainty, sometimes to our detriment, maybe sometimes to our advantage.

I’ve been working with ROAS quite a lot, and trying to really get my teeth into it, and my brain around its under-the-surface complexity. For most businesses today, ROAS is useful, but it is not as simple as it looks.

This is where I ask you to add something in the comments! What metrics are stuck in your craw this week? Do you think I spend too much time trying to become certain about uncertainty? Let me know!

# Your Classic New Year’s Post

2017 was a huge year for me and the Doc and the kids!

My kids turned three and one, and are starting to become little people with their own opinions and positions – they are also developing a complex relationship with one another which I had for some reason totally failed to expect until it came into sudden and sharp focus.

My wife, the Doc, has received tenure! Associate Professor no more, she has a new name badge on her office door and a continued bright professional future. I am so proud of her, even as this accomplishment is staggering for me to consider: she’s spent practically the whole of her professional life working toward this goal. When she started graduate school, ostensibly the first step toward tenure, I was still an undergraduate. This is the culmination of more work than all of the work I’ve done. She’s amazing.

I moved into this new role at WordPress.com – from leading a Happiness Team, to now this more data focused role with our (still pretty new!) Marketing team. The job title is Data Analyst but, like any job at a startup, I have plenty of hats. Adjusting my mindset from leading humans to writing SQL has been a challenge, but a great opportunity for growth as well.

2018 is shaping up to be an exciting year: it will mark my fifth anniversary at WordPress.com and Automattic, which I’ll be able to celebrate by taking a 3-month paid sabbatical in late summer / early fall.

I think, in 2018, I would like to try to spend some time leaning into being more of a light bulb: I’ve found success in focus, in great focus and goal-setting and the fairly standard practices there.

I’ve always felt a certain drive to be significant, to try to be a Big Fish in whatever capacity I can – in 2018 I think I’ll try to be OK as a Small Fish. Doing my work well, contributing thoughtfully without fanfare. Rather than shouting, focus on listening. Try to really embrace the role of the student. That sort of thing.