Incrementality Testing: How to Uncover the Real Impact of Your Campaigns

If you’re an eCommerce brand spending more than $30K/month on advertising and you’re not running incrementality tests, you’re leaving 15-30% of profitability on the table.

We all know we can’t trust Meta, Google, and TikTok’s results 100%.

It’s like asking a kid to grade their own homework and expecting them to tell the truth. In most cases - like with brand campaigns - they inflate their results.

Luckily, there’s a way to know the real impact of your campaigns and make decisions based on data and experimentation.

Incrementality tests.

What is incrementality

I’ve talked about incrementality before in past posts - How to Turn Meta Into Your Profitability Machine, 7 Habits of Highly Profitable eCom Brands, and Your ROAS Doesn’t Mean Shit.

But I feel like I’ve never gone deep enough.

So let’s start from the beginning.

Incrementality refers to the true impact of your marketing campaigns.

And it answers the question: “How many extra sales did my marketing effort generate that wouldn’t have happened without it?”

To put a number on not knowing the answer: if you spend €240K a year on advertising, it can mean €60-96K a year in bad investment decisions.

In an ideal world, this information would be in the ad platforms’ dashboards, but we know we can’t trust their data.

And not because it’s their fault, but because the system has structural limitations.

Meta and Google can’t measure what really matters

In an effort to provide clarity, the platforms report their results the best way they can and let you configure when campaigns get credit for a sale and when they don’t.

Let’s look at an example.

Meta has 3 attribution settings:

7-Day Click: The platform takes credit for a sale when the user clicks its ad and buys within the next 7 days.
1-Day Click: The platform takes credit for a sale when the user clicks its ad and buys within 24 hours.
7-Day Click / 1-Day View: The platform takes credit for a sale when the user clicks its ad and buys within the next 7 days, or when the user saw the ad, didn’t click, but bought within the following 24 hours.

The system works, but it has one fundamental limitation: it can’t tell correlation from causation.

Platform attribution says “this person saw/clicked the ad and bought. Therefore, I’m responsible for the sale.” But it ignores whether that person would have bought anyway.

The clearest example is Google brand campaigns.

In many cases, when someone searches for your brand on Google, they already intend to buy, but they click the paid ad because it’s the first thing that shows up.

Google takes credit for that sale even though it wasn’t responsible for generating it, which makes your reports look spectacular while your real sales don’t go up.

Don’t expect the platforms to fix it

This isn’t a technical limitation at Meta, Google, and TikTok that gets solved with a new feature - it’s structurally impossible for them to measure the true incrementality of their campaigns, and there are several reasons why:

They can’t track people who haven’t seen their ads
They can’t account for the effect of other platforms
They can’t count the same person across different devices
They’re increasingly limited by privacy issues like GDPR, users running AdBlockers, and the launch of new operating systems like iOS
It’s not in their best interest to show true results

If you want to know the real impact of your marketing efforts, you need to take matters into your own hands and start running incrementality tests.

The only way to know your real numbers

Now that we understand why the platforms can’t (and don’t want to) measure the real incrementality of their campaigns, let’s talk about how you can.

There are several methods for testing incrementality, but the most effective and accessible for eCommerce is “Geo Holdouts”.

In Geo-Holdout tests, geographic areas (usually provinces) are split into two similar groups:

Control, where nothing changes
Variant, where the campaign or marketing channel you want to measure is increased, decreased, or paused.

And after several weeks, you measure how sales have evolved in the two regions.

The difference between them is what reveals the true impact of your effort.

The control group can take several forms depending on the method you use, but these are the most common:

Specific locations: You select specific geographic areas as the control group to compare against the treatment areas.
Synthetic control by amalgamation: You build a control group by combining several regions with different weights to match the treatment group (30% of A, 50% of B, and 20% of C)
Algorithmic synthetic control: You use statistical models to simulate how the treatment group would have behaved, based on a control group.

Let’s go through an example using specific locations so you can see it more clearly.

Note: This is a simplified example to illustrate the concept. In practice, we work with more sophisticated statistical models.

Incrementality testing in action

On Meta we spend $40,000 across 2 similar cities, Madrid and Barcelona ($20K in each).

Each city does $100,000/month in revenue.

In its reports, Meta says we’re generating $50K in revenue in each region. That’s a ROAS of 2.5.

To validate this, we run an incrementality test.

We pause Madrid for a month and watch how its revenue evolves.

After 30 days, we see Madrid has gone from $100,000 in revenue to $60,000, while Barcelona is still doing $100,000.

This test tells us that Meta actually generates $40,000, not $50,000.

In other words, it’s taking credit for more sales than it should, and its ROAS isn’t 2.5, it’s 2.

Now we know the real impact of our Meta campaign, but what can we do with it?

Incrementality factor

To “counteract” the platform’s extra attribution (or in some cases, its under-attribution), we can apply what’s called the “Incrementality Factor.”

Formula: Real ROAS from the Test / ROAS Reported by the Platform

In our case: 2/2.5 = 0.8 or 80%

The next time you look at Ads Manager, you know Meta’s results need to be cut by 20% to reflect real incrementality and over-attribution.

This lets you keep spend under control and stop the platforms’ ROAS from fooling you.

How incrementality tests protect your profitability

For example, let’s say the most you can afford is a ROAS of 3, and you have a Google Branded Search campaign with a ROAS of 10 that your agency calls spectacular, but you don’t trust.

You run an incrementality test and realize this campaign has an incrementality factor of 20%. In other words, the ROAS that used to be 10 is actually 2.

What used to look profitable, you now know is losing money. That’s the power of these experiments.

One of our clients in apparel discovered that their Google Branded Shopping campaign, which reported a ROAS of 8, had a real incrementality of just 3.

Pausing it and reallocating that budget to channels with higher incrementality generated 15% more in net sales.

Life after your first incrementality test

After your first incrementality test, the real work is just getting started.

Just like CRO - which is where I come from - you only start seeing the real return on these practices after months and several experiments.

Running a single test is fine for validating a specific hypothesis and getting your feet wet, but marketing isn’t static: platforms change their algorithms, your audiences evolve, you develop new products, and new competitors show up.

Plus, you can keep getting more granular or testing new channels.

Here’s an example of a roadmap we put together for one of our clients:

Round 1: Incrementality by platform (Google, Meta, TikTok)
Round 2: Incrementality by campaign type (Google Branded Search, PMax, YouTube)
Round 3: Incrementality by creative: It might not seem like it, but creative can drastically change a campaign’s incrementality (HAUS example with Javy Coffee)
Round 4: We start over after 6 months

The value is in making experimentation part of your culture.

That way you make sure you’re not making decisions based on outdated data, but on recent experiments.

And if your eCommerce sells through other channels like Retail or Amazon, incrementality will be an essential tool for determining the real value your campaigns produce, not just on your website but across every point of sale.

Conclusion

Incrementality tests are where data-driven marketing, experimentation, and finance meet.

Bringing them into your operation won’t just make your efforts more efficient - it’ll give you the confidence of knowing every euro you spend is generating a real return.

It’s also true that the existing platforms are expensive and don’t go below $2,000 a month.

That’s why we developed an in-house methodology that lets us run these tests at no extra cost to our clients.

We know it’s the only way to get real clarity on the impact of our work.

And that without that clarity, both we and our clients run the risk of optimizing metrics that don’t translate into real growth.

If you’re managing a sizable budget and want to validate the real incrementality of your channels, we can review your specific situation and explore how these tests could apply.

Analyze your specific situation and build a sustainable growth plan.

BOOK A CALL

We’re looking forward to it!