Geo holdout test for Shopify brands: one design, end to end.

Is Meta making you money, or taking credit for money you were going to make anyway?

Every brand at this size has asked the question, and most have been answered by the platform being asked. Ads Manager reports what it attributes. Google Ads reports what it attributes. Analytics reports a third number and the store a fourth, a pattern the four revenue reports explains. None of them is an experiment.

A geo holdout is. It is the most accessible way to measure incrementality for a Shopify brand, because the outcome you need is already sitting in your order export: revenue by shipping address. This post walks through one design end to end, with the numbers a brand spending around $150,000 a month on Meta would face. For the wider menu of tests and when to pick each, start with incrementality testing without a data team. This post goes deep on a single design.

The design in one paragraph

Pick one channel and one question. Choose geographic units you can both target in the ad platform and assign from a Shopify shipping address. Use a long daily history to build a comparison, such as a synthetic control, that tracks your test markets closely before the test. Check power on that history before you commit. Turn the channel off in the test markets for at least one full purchase cycle and change nothing else. Then compare what the test markets did against what the comparison says they would have done, and divide the revenue difference by the spend you withheld. That is incremental ROAS, and a geo holdout is read by comparing its interval to breakeven.

Step 1: write down the question and the decision

Start with the decision the test is meant to change. Asking whether Meta is incremental is a weak question, because almost every channel with real spend is somewhat incremental and the answer will be yes. A better one is this: at current spend, does Meta’s incremental return clear our breakeven ROAS of 2.0, and if not, by how much should we cut? That version names the metric, the threshold, and the action. The breakeven ROAS calculator gives you the threshold from your own margin.

Write the decision rule down before the test starts. If the whole interval is above breakeven, keep spending and consider testing a higher level. If it straddles breakeven, trim in steps and retest, because a geo test measures the average return and the next dollar returns less than the average. If the whole interval is below, cut. Committing in advance matters more in geo tests than almost anywhere else, because the readout has enough moving parts to argue any result into any conclusion after the fact.

Also decide whether you are measuring the average or the margin. Turning Meta off in the test markets measures the average incremental return of the whole channel. The return on the next dollar is lower, as marginal ROAS explains, and measuring it directly needs a partial reduction rather than a shutoff, which in turn needs a larger or longer test.

Step 2: choose the geographic unit and the outcome

The unit has to satisfy two systems at once. The ad platforms must be able to target it, and your order data must be able to assign every order to it. In the United States that usually means states or metro areas, and in the UK postcode areas, because the regions give too few units. GeoLift’s best practices recommend the finest granularity you can target, such as ZIP codes or cities, because more units give the method more to work with.

The outcome should come from Shopify, not from the platform being tested. Export orders with the shipping postal code, map each code to your geographic unit, and build a daily series of net sales or order count per unit. Pick one before the test and stay with it. Order count is often less noisy than revenue because it ignores swings in basket size, while revenue is what the breakeven rule needs. If you choose orders, multiply by average order value in the readout and say that you did.

A few Shopify details matter. Exclude orders the ads cannot influence, such as wholesale or draft orders, if they are material. Decide how to treat returns, which arrive weeks later, before the readout rather than after it. And if a meaningful share of orders ship somewhere other than where the buyer lives, gifts in particular, note it as a known leak. It biases the result toward zero.

Step 3: match the markets

There are two respectable ways to build the comparison, and they reflect a real difference in what each approach protects against.

Synthetic control, the approach behind Meta’s open-source GeoLift, builds a weighted combination of untreated markets that tracks the test markets during the pre-period, then uses that combination as the counterfactual during the test. GeoLift’s methodology combines augmented synthetic control with generalized synthetic control inference. It is a practical default when you have a handful of candidate test markets and a long daily history.

Randomized paired designs, the approach behind Google’s Trimmed Match research, pair similar geos and randomly assign one of each pair to treatment. The Trimmed Match work states the problem plainly: the number of geos is small, they differ from each other a great deal, and trimming the most extreme pairs protects the estimate from a few unusual markets. Google’s matched markets library, built on time-based regression, searches for comparable groups of markets under an advertiser’s own constraints.

For a brand without a statistician, the choice of method matters less than the discipline around it. Match on the trajectory of the outcome, not on size. Keep your largest markets out of the test group if losing their ad-driven sales for six weeks would be commercially unacceptable. And look at the pre-period fit with your own eyes. If the comparison does not track the test markets closely before the test, nothing after it can be trusted.

Step 4: size the holdout and the pre-period

Holdout share is a trade between precision and exposure. A larger holdout makes the effect easier to detect and puts more revenue at risk. As a planning starting point, not a published standard, we begin with test markets holding about 20% of revenue and let the power analysis move the number up or down.

The cost of the holdout is worth computing, because it is usually smaller than people fear and it has a counterintuitive shape. Withholding spend saves the spend and loses the contribution that spend would have bought. So the net cost is withheld spend times incremental ROAS times contribution margin, minus the withheld spend. At a 50% margin and an incremental ROAS of 3.0, withholding $42,000 costs about $21,000 of contribution. At 2.0 it costs nothing. At 1.3 it saves about $14,700. The test is most expensive when the channel is strongest and pays for itself when the channel is weakest, which is exactly when you most need to know.

For the pre-period, GeoLift’s best practices ask for stable history of at least four to five times the test duration, at minimum 25 pre-treatment periods across 20 or more geographic units, and ideally the last 52 weeks. Daily data is strongly preferred over weekly. A six-week test therefore wants at least 24 to 30 weeks of clean history, and a full year is better because it shows the method how your markets behave through a season.

Step 5: check power before you commit

Power is the probability that a test detects an effect of a given size when the effect is real. Skip it and you risk the worst outcome a test can have: a result that finds no significant lift when the channel was working, which then gets quoted for a year. GeoLift’s methodology gives the same warning and provides power calculators for test length, investment, and number of test markets, run against your own history.

You can do a first pass on paper. The planning formula in the incrementality testing guide puts the minimum detectable effect at about 2.8 times the coefficient of variation, the week-to-week standard deviation of revenue in the test regions divided by its average, times the square root of two divided by the number of weeks. With an 8% coefficient of variation, four weeks gives a minimum detectable effect of about 15.8% and six weeks about 12.9%.

Now estimate the effect you expect. In this illustrative brand, Meta spend is 10% of revenue. If you plan on a cautious incremental ROAS of 1.5, well below the 3.4 the platform reports, turning Meta off should remove about 15% of revenue in the test markets. Fifteen percent is below the four-week threshold and above the six-week one, so the design runs for six weeks. Had the expected effect fallen below both, the right move would be to reduce noise, by adding markets or improving the match, rather than to run longer and hope.

GeoLift’s guidance also asks the test to cover at least one purchase cycle, with a minimum of about 15 days on daily data or four to six weeks on weekly data. For most Shopify brands selling considered purchases, four to six weeks is the realistic floor.

Step 6: run it without breaking the account

Most failed geo tests fail in execution, not design. The goal during the test is simple to state and hard to keep: the only difference between test and control markets is the channel you switched off.

• →Exclude the test markets rather than pausing campaigns. Pausing whole campaigns disrupts learning and moves budget in ways you do not control. Location exclusions keep the structure intact. Reduce budgets by the test markets’ share at the same time, or the money flows into the control markets and inflates them.
• →Check the location setting on the channel you are testing, and on every other channel that could leak into the test markets. Google Ads, for example, can target people by physical location, by locations they have shown interest in, or both. Interest-based targeting lets ads reach people inside your test markets who searched for somewhere else, so use presence-based targeting for the test, and check the equivalent setting on every other platform before launch.
• →Hold everything else constant across both groups: email, SMS, influencer posts, promotions, and other paid channels. GeoLift’s best practices make the same point about keeping local and national media steady. A national email is fine because it lands on both groups. A regional promotion is not.
• →Freeze the calendar. Do not launch across a major sale, a product launch, a price change, or a stockout that hits some regions harder than others.
• →Monitor delivery, not results. Check daily that spend in the test markets is actually zero and that control-market spend matches the plan. Do not peek at the lift and stop early when it looks good or bad. Early stopping manufactures false results.

Step 7: read the result against breakeven

Here is an illustrative readout for the six-week Meta holdout designed above. The numbers are arithmetic, not a client result.

The point estimate says Meta returned $2.10 of revenue for every dollar in the test markets, just above the 2.0 breakeven. The interval says the truth could sit anywhere from 1.30 to 2.90. That straddles breakeven, so by the rule written down in step 1 the verdict is to trim in steps and retest. Do not switch the channel off, and do not scale it.

Notice what the interval does to the conversation. A team reading only the point estimate would declare Meta profitable and move on. A team reading only the significance test would declare it proven and perhaps raise the budget. Both would miss that the average return barely clears breakeven, which means the marginal return, the one that governs the next dollar, is very likely below it.

The last two lines are the reusable output. Meta’s own reporting said 3.40 and the test says 2.10. An incrementality factor of about 0.62 lets you discount platform-reported ROAS in the months after the test, as long as the account, the spend level, and the mix stay broadly similar. It decays. Retest when any of them move.

What the common advice gets wrong

The common advice is to run a geo test and check whether the lift is statistically significant. That frames the test as a trial of whether the channel works. It is the wrong trial.

Almost every paid channel with real spend produces some incremental sales, so a well-powered test will usually find a lift different from zero. The commercial question is whether the lift is large enough to pay for the spend, which is a comparison against breakeven, not against zero. A channel can post a significant lift and still lose money. In the readout above, the lift is clearly different from zero and the channel is still only borderline profitable.

The second error is treating one test as permanent truth. A geo holdout measures one channel, at one spend level, in one season. It is the most trustworthy number you will have about that channel, and it has an expiry date.

Failure modes that quietly ruin tests

• →Spillover. People in test markets still see ads, through interest-based location targeting, travel, or imprecise location signals. Google describes location as inferred from IP address, device location, and other signals, which is good but not perfect. Spillover makes the channel look less incremental than it is.
• →Markets too small. With few or tiny test markets, ordinary day-to-day noise swamps the effect. The power check catches this if you run it. The fix is more markets or a better match, not a longer wait.
• →Calendar events. A holiday, a sale, a heatwave, or regional news that lands unevenly across test and control markets will be read as lift. Check the calendar before launch and again before the readout.
• →Budget leaking into control markets. If the excluded test-market budget flows into control markets, the comparison is inflated and the lift is overstated.
• →Contaminated outcomes. Wholesale orders, marketplace orders, or one large business order can dominate a small market’s week. Clean the series with rules you set before the test.
• →Moving the goalposts. Changing the outcome metric, the window, or the markets after seeing the data turns an experiment back into an opinion.

What to do this month

It is mid-November. Do not start a geo holdout now. Black Friday, Cyber Monday, and December gifting will distort both groups unevenly, gifts will ship to addresses far from the buyer, and a holdout during peak puts the most revenue at risk at exactly the wrong moment. Use the month to get ready instead, so a clean test can launch in the second half of January.

1. Export a year of Shopify orders with shipping postal codes and build the daily revenue series by geographic unit. This is the slowest part, and it needs no decisions yet.
2. Choose the channel and write down the question and the decision rule, including breakeven ROAS at your real post-promotion margin.
3. Run the power check on the history you just built, and pick markets, holdout share, and duration from it.
4. Note every regional promotion, stockout, and anomaly in that history so the pre-period can be cleaned, and plan the January calendar to avoid new ones.
5. Make sure your analytics are clean enough to cross-check the result. If GA4 is part of that cross-check, our GA4 and measurement work is the place to start.

A geo holdout is the closest thing a Shopify brand has to a controlled experiment on its own advertising. Designed carefully, it replaces an argument between four dashboards with one number and an honest range around it. If you want an independent read of your account before you design one, start with the free audit.