I ran an experiment… Now What?

Introduction

It's no secret that the key to driving personalized marketing is customer data. Time and time again, we've shared real-world examples of how through Data Activation, marketing teams from leading brands like Warner Music Group and PetSmart are able to unlock the full potential of their data to create amazing experiences that connect with consumers.

Recently, we've been hearing from our customers that they needed more help. They could run more experiments than ever, but they were struggling to understand how well those experiments were performing. They needed better tools to learn what was working and to apply those lessons to future campaigns and experiments. And they needed them to work without complex SQL queries or waiting in a long backlog for some time with their data analysts.

Well, we've been listening. That's why I'm thrilled to unveil Hightouch Performance, a visual set of tools that enables any marketer to explore the data and performance of their audiences and experiments without SQL or submitting a single analytics ticket. This new capability is now available in beta as part of Customer Studio for interested customers.

In this post, we'll explore why audience measurement is so valuable to modern marketing teams and how you can use Hightouch Performance to accelerate growth and experimentation in your own marketing strategies.

The Measurement Challenge in Modern Marketing

The modern marketing and advertising landscape is booming. There are more marketing channels (email, sms, etc.), "walled garden" advertising platforms (Facebook, Google, etc.), and media networks (Amazon, Walmart, etc.) available now than ever before. And thanks to the rise of the modern data and marketing stack, it's never been easier to spin up new channels and experiment with new tactics.

All of this freedom to experiment has a downside that simply cannot be ignored. With so many channels and touchpoints in flight, today's channel plans look something like the San Francisco MUNI transit map that I've come to know and love... In this complexity, marketing teams struggle to understand which of their many investments are responsible for increased performance–and which they can cut to save money.

To answer those questions, most marketers turn to attribution modeling to provide insights into which specific channels and touchpoints caused down-funnel conversions. Unfortunately, attribution modeling often under-delivers as a measurement solution.

Each of the many available attribution or measurement methods organizations use today -Media Mix Modeling, Sales Lift, Last Touch, First Touch, or Linear Attribution- take different approaches to form a directional indication of how well individual channels and campaigns perform. Their ability to represent relationships between the tactics you run and the outcome your business gets is limited to the quality of the available data. If a UTM parameter is stripped out, a click is missed, or a cookie is deleted, your audience's traceability is irrevocably compromised. This means attribution experiments often give incomplete results and don't really tell you whether your treatment group contributed more revenue than your control audience.

"The pursuit of a perfect attribution model would assume that you would not only be able to model every permutation of a path to purchase, but you also would be able to quantify each component’s value in driving a visitor to purchase. With a nearly infinite number of options, that simply isn’t possible."

Attribution exercises are important and worth doing, but no matter how much your organization has invested, I can assure you that there is no golden answer out there waiting to be uncovered. Yes, this even applies to your new CMO, who promised to implement more modern marketing attribution efforts this year.

So if most attribution modeling delivers limited value, how did we set off on our path to provide the necessary insights to evaluate and iterate on experiments at scale? We chose a different and complementary path: audience measurement.

What is Audience Measurement?

Unlike attribution modeling, the goal of audience measurement isn't to understand channels or touchpoints in isolation. In fact, it doesn't even need to know what types of campaigns are being run–and that's a good thing.

It's a simple yet powerful analysis of the performance of a specific cohort of users over a period of time. This time-based analysis unlocks the insights that enable marketers to see how their marketing efforts may have impacted certain KPIs like Conversions or Average Order Value.

The simplest example of this can be seen below, where an e-commerce business wanted to better understand how various channel-specific treatments impacted their marketing funnel as a whole. In this case, a cart abandon audience was created in Customer Studio with three "Splits" or--random experiment groups– which are then automatically synced to their email platform

1. The control group was sent the standard cart abandonment email
2. Treatment 1 was sent an email personalized to the items they left in their cart
3. Treatment 2 was given the same personalized email but incorporated a personalized subject line as well

Once launched, the team could use the new Hightouch Performance feature seen below to measure relevant KPIs for each group to determine a champion experience.

To no one's surprise, users in Treatment 2 outperformed the other treatments and ultimately drove more revenue at the end of the day for the business.

With this audience performance view, the team has confidence that regardless of what may have shown in their email platform from direct conversions (users clicking emails), the impact was measured as a whole - even counting users who may have converted organically at a later point in time. Additionally, with these insights directly in the same platform where all their audiences and experiments are being managed, the team can move more efficiently to build, measure and continuously optimize future campaigns.

How Audience Measurement Enables Growth and Rapid Experimentation

Almost every growth team today understands experimentation is the best way to learn quickly and unlock sustainable growth. So what differentiates merely good teams from great teams? It's often the quality and speed of their experiments. Any team can run a single-channel experiment and measure it relatively quickly in downstream destinations like Facebook or Iterable, but a great team will use audience or cohort-based analysis to explore much more complicated and meaningful experiments.

Let's say the same e-commerce company discussed above wanted to take their experimentation to the next level and test new marketing channels. They will again create three splits within their cart abandonment audience for this experiment. This time, however, rather than sending all of them to only their email platform, each group will be sent to a different combination of channels.

1. The Control group will act as a holdout test and receive no additional remarketing
2. Treatment 1 will receive a personalized email similar to the one from the earlier test
3. Treatment 2 will also receive the personalized email, however, they will now additionally be targeted with paid advertising campaigns via Google Ads

Once the campaigns have launched, with audience measurement available directly in Hightouch, the team can quickly and efficiently measure the impact of the various treatments on each split. Diving into custom metrics defined in Hightouch, like LTV, the team quickly identifies how adding one or many channels impacts overall retargeting performance - and if doing so positively impacts their ROAS.

With Hightouch, our team can effortlessly experiment on and measure how certain cohorts of users are performing. This empowers us with valuable insights to understand the impacts of different campaign tactics across our growth channels

Kishan Antala

Product Manager

•

Morning Brew

A caveat: of course, we're aware audience measurement has limitations as well. There may be other factors that affect a given audience that we don't totally understand and can't see with this approach to performance measurement. It can also be challenging to weigh different factors' relative contributions and effects in a multi-channel experiment. The truth is that all attribution methods suffer from this issue, while the benefit of audience measurement is how comprehensive it is, how fast it is, and its ability to correlate real business metrics (like order value) to your campaigns. Although imperfect, we know it's an essential and under-utilized tool for marketing teams.

Wrapping Up

Modern marketing journeys are complex - they incorporate many channels with even more touchpoints, and they're not getting any simpler. As organizations continue to invest in launching new channels and more coordinated omnichannel experiments, they will need multiple measurement methods to drive insights that help them optimize campaigns. By investing in building out analytical features like Performance, we are taking another step on our journey towards building a Data Activation platform that enables every member of an organization to explore, build, manage, activate, and now measure customer data at scale - all with the data residing in the cloud data warehouse.

Interested in getting started? The Performance feature, currently live for beta customers, will be available for all advanced Customer Studio customers next week at no additional cost. If you're interested in learning more about this feature or understanding how it can help your team with experimentation or campaign evaluation, book a demo here.

Onwards,

Tejas