Incrementality testing just became performance marketing's most trusted skill. Most teams still can't do it.

Share
Incrementality testing just became performance marketing's most trusted skill. Most teams still can't do it.
Photo by Imagine Buddy / Unsplash

Here's an odd fact about performance marketing right now. The measurement method marketers trust the most is also the one fewest of them can actually run properly.

A recent survey of 500 senior US marketing decision-makers found that incrementality testing (more on what that means in a second) is now the most trusted way to measure marketing, ahead of media mix modeling and nearly twice as trusted as the numbers built into Google, Meta and TikTok's own dashboards. At the same time, a separate industry report found that three out of four marketing leaders say their core measurement methods aren't working well.

So marketers trust the method. They just aren't confident they're doing it right. That gap is basically the whole story here.

Quick definition: what is incrementality testing?

Incrementality testing answers one simple question: did this ad actually cause a sale, or would that sale have happened anyway?

You do this by holding an ad back from part of your audience (a "holdout" group) and comparing them to the group that did see the ad. The difference between the two groups is the real, "incremental" effect of the ad. It's the only major measurement method that actually proves cause and effect, instead of just noticing that two things happened around the same time.

Why "last-click" finally lost people's trust

For over a decade, performance marketing leaned on something called last-click attribution. That means giving all the credit for a sale to whichever ad or link the customer clicked right before buying.

It was popular because it was simple, not because it was accurate. It never really proved the ad caused the sale. It just noticed which ad happened to be closest in time.

That trust has now collapsed. Only about 1 in 5 marketers say last-click gives a fair picture of an ad's real impact, and about three-quarters are actively moving away from it. Meanwhile, most marketing leaders now believe at least 10% of their ad budget is being wasted because of bad measurement, and some think it's closer to 30%.

Incrementality testing has taken over as the trusted alternative. Adoption backs this up: around half of US brand and agency marketers now say they run incrementality tests, up from a niche practice just two years ago. In retail media specifically, over 70% of advertisers now say incrementality is their single most important number to track.

That's the encouraging part. Here's the catch.

Running a test and running it well are two different things

Saying you "do incrementality testing" and actually doing it correctly are very different levels of effort, and most teams doing the first haven't reached the second.

A proper test (usually called a "geo holdout") means picking a group of markets or users, deliberately not showing them an ad, and comparing what happens to a similar group that does see it. Done properly, this takes real analyst time to set up fairly, several weeks to get a trustworthy result, and enough ad spend involved to make the temporary loss worth it.

Because of that cost, how well a company does this still depends mostly on company size, not marketing skill:

  • Under about $5 million in revenue: the cost of running a clean test usually isn't worth the answer, so most brands skip it.
  • $5 million to $50 million: brands start running their first real tests, since the budget can handle a temporary 5–10% cutback for a few weeks.
  • Above $150 million: ongoing testing becomes close to standard practice.

In between those tiers, a lot of what gets called "incrementality testing" is really just reading a report the ad platform generated about its own performance. That's not the same thing as an independent test, and it's worth knowing the difference.

The catch nobody's talking about: who owns the tools

There's a second, quieter problem building here. The tools used to measure ad performance are increasingly owned by the same companies selling the ads.

Google recently released a free tool that helps run these geo tests and feeds results into Google's own measurement software. That's genuinely helpful for smaller teams who couldn't otherwise afford this kind of testing. But Google has also reportedly linked its own sales targets to how many advertisers use that tool. Meanwhile, Meta has quietly shut down its own competing open-source measurement tool.

In plain terms: the same companies selling you ads are becoming the ones grading how well those ads worked. That's not necessarily dishonest, but it's a conflict of interest worth having a second opinion for.

A simple 3-question test for your own team

Before your team claims it "does" incrementality testing, ask these three questions honestly.

1. Is at least one test run independently, without relying on the platform's own tool? If every result comes from Google or Meta's own reporting, that's not an independent measurement. It's the seller grading its own homework.

2. Is the test actually big enough and long enough to trust? A test with too few markets or too short a timeframe can still spit out a number, it just won't be a reliable one. If nobody on the team can explain why the result is trustworthy, treat it as a guess.

3. Does the team actually change spending when a test gives a bad answer? This is the real test. A lot of teams run the experiment, don't like what it says, and quietly keep spending the same way anyway. That's not a measurement problem. That's a decision-making problem hiding behind a measurement program.

Most teams that say they "do" incrementality testing would fail at least one of these three questions.

What this means for your career, not just your dashboard

Here's the good news: running the actual test has gotten easier. Tools have improved a lot in the last two years.

What hasn't gotten easier is knowing which questions are worth testing, designing a fair test, and having the confidence to move real budget based on an answer nobody wanted to hear. That combination, some statistics knowledge, healthy skepticism of platforms, and the nerve to argue with a stakeholder who loves a channel, is a rare skill. Most performance marketing training never really teaches it.

That skill is going to matter more, not less, as ad platforms get more automated and harder to see inside. The people who can design and defend a clean, honest test will become more valuable than the people who are simply good at clicking buttons inside a platform's dashboard.

Three things to actually do this quarter

Run one independent test on whichever channel gets the most unquestioned budget. Not your smallest channel, the one nobody's allowed to challenge.

Stop treating a platform's own report as proof. Use it as a second opinion, never as your only opinion.

Decide in advance what you'll do if the test says something you don't want to hear. If your honest answer is "probably nothing," it's worth knowing that before you spend the time running the test at all.

Performance marketing spent the last decade optimising against numbers that were never really true. Incrementality testing is the best shot the industry has at optimising against numbers that actually are. Most teams haven't caught up to what they already say they believe. The ones who close that gap first will have a real advantage, not because the method is a secret, but because doing it well still is.

Read more