What Is Incrementality? 2023 Guide to Testing Incrementality

Understanding the effectiveness of specific advertising and marketing campaigns helps determine which methodology best boosts sales, revenue, and profit. Yet, in today’s increasingly complex and rapidly changing digital marketplace, how do you know which programs and touch points contributed to a sale? Can an online ad really contribute to a sale?

With the advent of digital marketing, return on ad spend (ROAS) became a simple core metric calculated by dividing total ad spend by total revenue. However, ROAS fails to consider the bigger picture.

Incrementality goes deeper than ROAS. Incrementality tests compare control groups to help marketers determine which ads work and which don’t. This process guides them to spend their marketing budget on campaigns that deliver the best results and return on investment (ROI).

We outline the core concepts of incrementality, covering what it is, what it can solve, and how to use it.

What is incrementality?

Incrementality is a methodology used to quantify the additional value or impact derived from a particular action or intervention compared to a baseline or control scenario. It involves measuring the incremental effect or benefit generated by implementing a specific strategy or making a targeted change.

Incrementality testing embraces a more evidence-based, data-driven, scientific approach using A/B test groups. For example, control group A doesn’t see an online ad, and group B does. If group A spends the same amount on a product or service as group B, the ad spend didn’t contribute to conversions and additional revenue as planned.

Those dealing with larger, more complicated marketing campaigns might need multivariate testing. It compares more variables than simple A/B or holdout testing and reveals telling information about how these variables act with one another.

Why is incrementality important?

When you have a marketing campaign with many moving parts, evaluating how each component does or doesn’t contribute to its overall success can be challenging.

For a paid online advertising campaign, incrementality might reveal how much the ad affected the response volume and what you would have received through organic results alone.

Incrementality testing lets you examine each channel, tactic, and  component that contribute to increased sales or signups.

Benefits of incrementality testing

Incrementality testing helps your marketing team determine which parts of a campaign are giving it momentum, so you can invest your budget more successfully.

Incrementality testing also helps:

• Show if two marketing investments are cannibalizing each other
• Determine which media channels help hit a certain key performance indicator (KPI)
• Predict what will happen if you introduce a new tactic or channel
• Predict what will happen if you adjust or reallocate your marketing spend
• Identify the positive, negative, or neutral impact of any marketing effort
• Prevent wasted ad spend

What does incrementality help answer?

Incrementality helps identify the lift marketing and advertising campaigns provide above native demand (baseline sales). This incremental lift provides data on sales and revenue attributed to a marketing or advertising program. In short, it answers the question, “Did this program increase the baseline demand for a product or service?”

You can apply incrementality testing to a variety of products and services. For example, let’s say you sell kombucha in retail stores and introduce a new flavor.

Some customers would have visited a store and purchased the new flavor regardless of your brand’s ad. But you don’t know whether viewing an ad will convince them to go to a store and purchase the new flavor. An incrementality test can help answer this question with a control group analysis.

In another example, let’s say you sell computer monitors and decide to increase your warranty period from one to three years to boost sales. An incrementality test can help determine if this tactic worked as planned.

How can marketers test and calculate incrementality?

You can implement a basic incrementality test by outlining all the test factors and developing an implementation plan. It takes some work upfront, but this is necessary to produce accurate results and best demonstrate whether your advertising efforts worked.

Incrementality test components

Conducting an incrementality test is like conducting almost any scientific test. You develop a hypothesis, a testing method, data collection, and a results analysis—all yielding your final conclusion.

The main components of an incrementality test include:

1. Define your goals. What are you trying to achieve with your marketing campaign? What vital business KPIs do you wish to explore?
2. Segment your audience. Decide which target audience is best suited for this type of marketing and who your control group should include within the audience.
3. Launch your test. Decide on a test length and launch it—preferably when your calendar is clear so you can fully immerse yourself in the experiment.
4. Analyze the data. Compare the data collected from the test and control groups to see if there’s an incremental lift based on a specific KPI.
5. Put insight into action. Take the insights from your data analysis and apply them to your marketing efforts for maximum impact.

What are the three stages of an incrementality test?

The three main stages of an incrementality test include preparing the separate test and control groups, introducing test one to a new variable, and measuring the test results.

1. Preparation

Identify and split an audience into A and B test groups that are statistically equivalent, are properly segmented, and have the same applied testing time. One group is the test group, and the other is the control group.

For example, a technology retailer might want to test the possibility of running display ads for the first time. It might target an audience of small business buyers, dividing them evenly into one group that sees the test display ads and one that views a typical marketing campaign.

2. Intervention  

An intervention exposes one group (the test group) to a new variable, such as an ad for a new or enhanced product or service, while the control group doesn’t see it.

The test group seeing new display ads and the control group looking at a typical marketing campaign should have enough media exposure for a difference in reactions to become apparent. The retailer might look at which approach is more effective in prompting viewers to purchase specials on computing tablets.

3. Measurement

Analyze the performance of the test and control groups, pre- and post-intervention, to determine the incremental impact of the intervention and whether it created an incremental lift.

Assuming the intervention group and the control group are equally split, calculate lift and incrementality using these formulas:

Lift formula

Lift = Test (conversion rate) – Control (conversion rate) / Control (conversion rate)

To calculate a conversion lift, you’d express the difference between the observed conversion rate of an intervention group and the baseline conversion rate of the control group as a percentage of the control rate.

So, if the conversion rate of the technology test group is 10% and the conversion rate of the control group is 8%, the conversion lift is (10 - 8) / 8 x 100 = 25%.

This indicated a 25% increase in conversion rate as a result of the display ads.

Incrementality formula

Incrementality = Test (conversion rate) – Control (conversion rate) / Test (conversion rate)

This incrementality formula helps measure the true contributions of a media channel, campaign, or ad to the business outcome—whether it involves leads, sales, or some other metric. You calculate the conversion difference between the intervention group and the control group.

For example, if a test group exposed to an ad had a 1.5% conversion rate and a control group without exposure had a 0.5% conversion rate, you’d have an incremental conversion of (1.5% – 0.5%) / 1.5% = 66.7%.

This indicates a significant 66.7& increase in the conversion rate that can be attributed to the display ads.

How to measure the impact of incrementality

Marketing mix modeling (MMM) and multitouch attribution (MTA) provide two ways to measure the effect of a marketing campaign on sales, profits, conversions, and any other desired outcome.

Marketing mix model

A marketing mix model uses a top-down approach to measurement, examining how each business aspect contributes to a final outcome. It often uses a statistical model to understand the overall impact of a business’s different marketing efforts on sales, conversions, and other results.

MMM doesn’t provide granular data (i.e., tracking individual users at specific touch points), but it does use aggregated campaign or channel data to provide big-picture information about marketing activities.

Use this information to determine the marketing mix that will yield the best results.

Multitouch attribution

A multitouch attribution model uses a more detailed, bottom-up approach to carefully measure every consumer’s media exposure and map a possible outcome.

It can help you assess the relative importance of each touch point on the customer’s journey, providing a huge amount of data to fine-tune or overhaul your marketing efforts.

Multitouch attribution is a far more granular method than MMM, providing more data on every channel and step in a larger marketing campaign. You can combine it with incrementality measurement to facilitate data-driven decision-making.

Incrementality and ROAS

Return on ad spend (ROAS) was once the most important metric for spending on advertisements (paid media). ROAS is calculated by dividing ad revenue by cost and provides a quick, easy-to-understand formula. However, it lacks the more detailed marketing measurements of incrementality. In some respects, incrementality takes over where ROAS leaves off.

While ROAS reveals the total amount of revenue offsetting the cost of ads, incrementality can show the number of conversions won by a campaign. It reveals how effective a campaign is and what the sales revenue would have been without the campaign. The detailed information it provides about individual user segments allows you to create more personalized marketing efforts.

Next steps after testing for incrementality

Once you’ve tested for incrementality, consider other costs affecting your bottom line, what it takes to acquire new customers, and other outcomes.

Marginal costs

Marginal costs are the increase or decrease in dollars to produce one more unit or to serve one more customer. In the world of marketing, marginal costs refer to the advertising spend it takes to get each new viewer.

The longer you chase additional customers through your marketing channels, the higher the acquisition costs. You must find a way to land new customers without driving up your marginal costs.

Cannibalization

Marketing cannibalization happens when a company introduces a new product or service that takes money or market share away from an existing one. When extending your brand with new offerings, take care not to make money on one hand while unintentionally losing it on the other. Cannibalization can also happen when you discount a product and get sales at the expense of other competing products in your inventory.

Let’s say a campaign designed to promote an affordable office chair unintentionally undercuts sales of your higher-priced ergonomic office chairs. This cannibalization could unfavorably impact your bottom line.

Incrementality test vs. A/B test: key difference

Since incrementality testing splits an addressable market or a segmented addressable market into two groups, control group A and test group B, you may wonder how it differs from A/B testing in digital marketing.

A/B testing—also known as split testing—compares two test versions of a creative (like a digital ad, an email, or a webpage) to compare variances in KPIs. These metrics include, but aren’t limited to:

• Impressions
• Click-through rate (CTR)
• Conversion rate
• Webpage bounce and exit rates
• Revenue lift

You can apply incrementality testing to a launched marketing campaign to measure that campaign’s impact on business metrics, like:

• Incremental revenue lift
• ROI
• New client acquisition
• Upselling

As with incrementality testing, A/B testing requires staff with the knowledge base and time to conduct meaningful A/B tests. A/B testing specialists on Upwork can help.

Incrementality example

Faherty is an American clothing company and beach lifestyle brand operating on the Shopify platform. A case study reveals that Faherty wanted to see if it had gone as far as possible with its Facebook and Instagram advertising. The brand also wanted to determine if it could drive aggressive growth through a diversified media mix.

Its incrementality marketing partner put cross-channel media reporting and an incrementality measurement into place to reveal the paid media contributions to online and offline transactions. It also:

• Deployed Facebook incrementality recommendations to better allocate money for the campaign, audience, and ad set
• Used a retargeting incrementality experiment to reduce the size of retargeting budgets without hurting performance
• Provided consolidated portfolio performance reporting to the finance and executive team

The incrementality testing and new marketing campaign helped Faherty increase sales by 53%, grow its new customer base by 33%, and experience other double-digit improvements.

Make incrementality tests work for you

Now that you’ve learned how incrementality tests can reveal the true impact of your marketing campaigns and help you better determine where your ROI comes from, you’re probably eager to perform your own incrementality test.

Doing so requires planning and the right tools to develop, launch, manage, and measure incrementality tests. You must also allocate marketing team resources with the expertise and knowledge to create and conduct tests that produce accurate, meaningful results.

If you don’t have available staff or staff with the appropriate experience and knowledge, you can consider staff augmentation. Staff augmentation increases your team’s talent capacity by filling the gaps with high-quality temporary workers. Upwork allows you to hire top independent professionals confidently using the world’s largest work platform.