A/B Testing Your Ad Setup: What to Test and How to Measure

You cannot optimize what you do not test. That statement applies to landing pages, game mechanics, and ad configurations alike. Yet many gaming publishers run their ad setup based on one-time decisions: they pick a layout, set some parameters, and never revisit them. The result: an ad configuration that was decent at launch and has been slowly degrading ever since.

A/B testing your ad setup is how you find the configuration that actually maximizes revenue for your specific site, audience, and traffic mix. Here is what to test, how to run valid experiments, and the common mistakes that waste your time.

What to Test

Not every variable is worth testing. Focus on the changes that move revenue meaningfully. Here are the six highest-impact variables for gaming sites, ranked by typical revenue impact.

1. Ad Density: 2 vs 3 vs 4 Units Per Page

Why it matters: Each additional ad unit on a page generates incremental revenue, up to a point. Beyond that point, more ads reduce CPMs (advertisers bid less when the page is cluttered), hurt viewability (more ads below fold), and degrade user experience (higher bounce rate, shorter sessions).

What to test: Run three variants:

• Variant A: 2 ad units per page (lean)
• Variant B: 3 ad units per page (moderate)
• Variant C: 4 ad units per page (dense)

What you will typically find: In our experience, the jump from 2 to 3 units often raises RPM by something like 25–40%, while the jump from 3 to 4 adds much less (often 5–15%) and starts to push bounce rate up. Treat those as rules of thumb; your test is the real answer. The sweet spot for most gaming sites is 3 units on game pages and 3–4 on content/editorial pages.

Measure: RPM (not just total impressions), bounce rate, average session duration, pages per session.

2. Ad Sizes: 300×250 vs 300×600 Sidebar

Why it matters: Larger ad sizes command higher CPMs because they offer more creative real estate for advertisers. But they also consume more page space, which can push game content or CTAs below the fold.

What to test:

• Variant A: 300×250 (Medium Rectangle) in the sidebar
• Variant B: 300×600 (Half Page) in the sidebar

What you will typically find: The 300×600 usually earns a noticeably higher CPM than the 300×250 in the same position (high-impact sizes typically clear at $4–10, against $2–6 for standard units). Illustrative US ranges from our experience; your rates depend on geography, format and season. However, on mobile devices, the 300×600 can dominate the viewport and feel intrusive. Test desktop and mobile separately.

Measure: CPM by size, viewability rate (the 300×600 must be at least partially in view to matter), CLS impact.

3. Ad Refresh Intervals: 30s vs 45s vs 60s

Why it matters: Ad refresh reloads the ad creative at set intervals while the user is on the page. Gaming sites benefit heavily from refresh because players often stay on a single game page for many minutes. Without refresh, you serve one impression per ad unit per pageview. With 30-second refresh, a unit that stays in view for, say, 10 minutes can serve up to 20 impressions.

What to test:

• Variant A: 30-second refresh
• Variant B: 45-second refresh
• Variant C: 60-second refresh

What you will typically find: Shorter intervals generate more impressions but lower CPMs per impression (advertisers discount frequently refreshed inventory). For most gaming sites, the revenue-maximizing interval lands at 30–45 seconds. Do not test below 30 seconds: many demand partners require at least 30, and CPM degradation outweighs the volume gain. Above 60 seconds, you are leaving impressions on the table.

Critical note: Only refresh ads that are in view. Refreshing an out-of-view ad breaks the refresh policies of many ad networks and SSPs. Use viewability detection before triggering each refresh.

Measure: Impressions per session, average CPM per refresh cycle, total RPM, SSP policy flags.

4. Interstitial Frequency: Every 3 vs Every 5 Game Plays

Why it matters: Interstitials (full-screen ads between game plays or levels) are high-CPM (typically $8–15) but high-friction. Too frequent, and users leave. Too infrequent, and you undermonetize engaged players.

What to test:

• Variant A: Interstitial every 3 game plays
• Variant B: Interstitial every 5 game plays

What you will typically find: Every-3 shows up to two-thirds more interstitials per game played (5 ÷ 3 ≈ 1.67). If it also shortens sessions by 10–20%, as it often does, the net gain in interstitial impressions per session shrinks to roughly 33–50%, and display impressions from the lost playtime go with it. Every-5 retains longer sessions and higher overall satisfaction. The net revenue difference is often smaller than expected: longer sessions with fewer interstitials can match shorter sessions with more.

Measure: Interstitial impressions per session, average session length, total revenue per session, user return rate (7-day cohort).

5. Floor Prices: ±20%

Why it matters: Floors directly control the bid/fill tradeoff. A floor that is 20% too high costs you impressions. A floor that is 20% too low costs you revenue on every impression that clears.

What to test:

• Variant A: Current floor
• Variant B: Floor increased 20%
• Variant C: Floor decreased 20%

What you will typically find: The optimal floor depends heavily on your traffic composition. For US-heavy traffic, higher floors often increase revenue (bids are strong enough to clear). For global traffic with significant Tier 2/3 volume, lower floors increase total revenue by recovering fill rate.

Measure: Revenue per 1,000 sessions, fill rate, average clearing price, revenue by geo segment.

6. Demand Partner Timeouts: 800 ms vs 1000 ms vs 1200 ms

Why it matters: In header bidding, you set a timeout: the maximum time Prebid waits for bid responses before sending the auction to the ad server. (Prebid.js defaults to 3000 ms, which is far too long for most gaming pages.) Longer timeouts collect more bids (more competition, higher clearing prices) but delay when ads render. Shorter timeouts load faster but miss slow bidders.

What to test:

• Variant A: 800 ms timeout
• Variant B: 1000 ms timeout
• Variant C: 1200 ms timeout

What you will typically find: For gaming sites, the best setting usually falls somewhere in the 800–1200 ms range. Each step up recovers some late bids, but the gain shrinks as the timeout grows, while ads render later and players may already be in the game or gone. Check the timeout rate per bidder: a partner that routinely misses 1000 ms is a partner problem, not a timeout problem.

Measure: Average bid price, timeout rate per bidder, ad render time, Core Web Vitals, total RPM.

How to Run a Valid Test

Traffic Split

Split your traffic 50/50 between control and variant. Use a server-side or tag-level split, not a time-based split (Monday = variant A, Tuesday = variant B), because day-of-week effects will contaminate your results.

If you are testing more than two variants (e.g., three refresh intervals), split traffic evenly across all variants. A 33/33/33 split requires more traffic to reach significance but avoids sequential testing bias.

Sample Size

You need a minimum of 100,000 sessions per variant for ad setup tests. Ad revenue has high variance: individual session revenue can range from $0 to $0.50+. Small sample sizes will show apparent "winners" that are just noise.

For a site with 500K monthly sessions testing two variants: you need approximately 200K sessions total, which means running the test for ~12 days.

Duration

Run every test for a minimum of 7 days to capture day-of-week effects. Ideally, run for 14 days. Ad demand varies by day: weekend CPMs differ from weekday CPMs, and a test that only captures Monday through Thursday will miss weekend dynamics.

The Right Metric: RPM

Do not judge ad setup tests by CPM alone. CPM measures revenue per 1,000 impressions. Page RPM measures revenue per 1,000 pageviews. A change that increases CPM by 20% but reduces fill rate by 30% is a net loss (1.2 × 0.7 = 0.84, a 16% drop), and RPM will show that.

The formula:

Page RPM = (Total Revenue / Total Pageviews) × 1,000

RPM accounts for fill rate, ad density, refresh behavior, and user engagement. It is the closest metric to "money in your account."

For gaming sites, an even better metric is revenue per session (or session RPM, revenue per 1,000 sessions), which accounts for session depth (pages per session). A change that increases RPM but shortens sessions might not improve total revenue.

Accounting for Seasonality

If your test runs across a CPM inflection point (e.g., late November into December, or late December into January), seasonal changes will dwarf your test effect. Either:

• Avoid testing during known inflection periods
• Normalize results against a holdout group that did not change
• Use a difference-in-differences approach to isolate the test effect

Common Mistakes

Testing Too Many Variables Simultaneously

If you change ad density, refresh interval, and floor pricing in the same test, you cannot attribute the result to any single change. Test one variable at a time. If you must test combinations, use a factorial design, but expect to need several times the traffic (a 2×2 design splits it four ways).

Insufficient Sample Size

The most common mistake. A publisher runs a test for 3 days with 20K sessions per variant, sees a 12% RPM increase in variant B, and declares a winner. A 12% difference is very likely within the noise range for 20K sessions. Run longer. Get more data.

Not Measuring User Experience

Revenue is not the only metric. A test that increases RPM by 15% but drives bounce rate up 25% and session depth down 30% is destroying long-term revenue for short-term gain. Always measure:

• Bounce rate
• Session duration
• Pages per session
• Return visit rate (7-day and 30-day cohorts)

If UX metrics degrade significantly, the revenue gain is not sustainable.

Declaring Winners Too Early

Statistical significance matters. Use a significance calculator; there are free ones available for A/B testing. A p-value below 0.05 (95% confidence) is the standard threshold. Do not declare a winner at 80% confidence: that means accepting a 20% chance of seeing a difference this large even when the variant has no real effect. Run a handful of tests at that bar and some of your "winners" will be noise. Also decide your sample size up front; checking every day and stopping the moment a result looks significant inflates false positives further.

Ignoring Segment-Level Effects

A test might show no overall effect but have dramatically different results by segment. Check results by:

• Device type (mobile vs desktop)
• Geography (US vs international)
• Traffic source (search vs direct vs social)
• Session depth (bounce visitors vs engaged users)

A variant that wins on mobile but loses on desktop might be worth deploying to mobile only.

Build a Testing Calendar

Structure your testing program across the year:

What to Do Next

Every untested assumption about your ad setup is a potential revenue leak. The question is not whether testing is worth it; it is which test to run first. Before you start, our guide to RPM, eCPM, and fill rate covers the metrics you will be reading, and you can see publisher results from gaming sites we work with.