Client-side vs server-side A/B testing Client-side vs server-side A/B testing

The verdict

At a glance

¹ VWO, who sell both: “a dedicated development team is required to run an end-to-end server-side test” (24 Feb 2026). · ² The mask carries a timeout, and that timeout is a cap for when the library fails to load, not a duration your page is normally blank. Both numbers are below. · ³ Averages across the pages the third-party-web dataset covers for each vendor; samples differ — Optimizely 25,015 pages, VWO 10,913, Kameleoon 3,585 (release 0.29.2, current as of 26 Jul 2026). · ⁴ WebKit, ITP 2.1, 21 Feb 2019. · ⁵ Mtrix's SDK writes the visitor-ID cookie with document.cookie, so on a plain CDN install Safari's cap applies to it too; the integration guide documents setting it from your own server instead. No dollar figures in the price row on purpose: this compares two architectures, and neither leading vendor publishes a price at all (both read 26 Jul 2026).

For the definition, the three separate decisions hiding inside the word “server-side”, and what Shopify Rollouts can and can't swap, read what is server-side testing. This page goes straight to the trade.

What deciding in the browser actually costs

The flash, and the mask that hides it

Flicker is what the architecture does, not a bug in anyone's tool. The browser renders what the server sent, the library rewrites the DOM, and the gap is the flash. AB Tasty's own description: “Flickering, also called FOOC (Flash of Original Content) is when an original page is briefly displayed before the alternative appears during an A/B test” — and it happens “due to the time it takes for the browser to process modifications” (How to Avoid Flickering in A/B Tests, 7 Apr 2017).

A browser gives you three answers. Load async, get the flash. Load sync, block rendering — Optimizely's own guidance is “You should not deploy your Optimizely Web Experimentation snippet as non-blocking” (8 Dec 2023). Or hide the page until the library is ready. That third option is the anti-flicker snippet, and it is where the honest argument lives.

Every one of these ships a mask with a fallback cap of 1,000 ms to 3,000 ms, as published in each vendor's own snippet or docs. Read them as ceilings, not as how long your page is dark — the number is what happens when the library doesn't arrive.

Two vendors say so themselves. Kameleoon's rule lifts “as soon as the application code loads, which typically takes less than 50 milliseconds,” with timeouts hitting only “about 2-3% of your visitors.” Adobe's doc is just as plain: “If the response from the server has been received and processed sooner, then the prehiding HTML style tag is removed as soon as possible.” VWO's live SmartCode shows the same shape in code — settings_tolerance arms a setTimeout that calls finish(), and finish() is also what runs on loading_failure. Same exit, two causes: the library lands, or it never does.

Optimizely's 3,000 ms belongs to the non-blocking install it explicitly tells customers to avoid; its default install blocks rendering instead, trading the blank page for a slower first paint. The field measurements below carry the duration argument — and they land well under every cap in this table.

The mask goes up for everyone, not just your test group

Google's published snippet — the load-bearing part is the first statement inside the function, and the 4000 passed in as the timeout:

<script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y; … })
(window,document.documentElement,'async-hide','dataLayer',4000,{'CONTAINER_ID':true});</script>

s.className += ' ' + y runs immediately and unconditionally on every page load — before the container loads, before any audience is evaluated, before anyone knows whether this visitor is in a test. Adobe's prehiding snippet appends its live style element on execution too.

VWO's is more careful than the others, and it deserves saying. Its current async SmartCode (v2.2, read live in vwo.com's own page source, 26 Jul 2026) inserts the overlay on execution — but hide_element() returns an empty string when performance.getEntriesByName('first-contentful-paint')[0] exists, so a page that has already painted is never masked. The older v2.0 in VWO's own PrestaShop plugin goes further: an async install fills the style element with nothing at all.

Where the mask is live, every visitor pays, on every page, including pages with no test running. And when the timeout expires you lose the test as well. Google, same page: “When the timeout expires, the page will unhide, the user won't see the new experience, and they'll just see the default content instead.” Blank page, then the control — counted in the experiment, never saw the variant. Sample contamination, described by the company that shipped the snippet, which warned on the same page: “Only install the anti-flicker snippet when necessary as it can impact on page performance.”

What the mask does to Core Web Vitals

Google's LCP spec excludes “Elements with an opacity of 0, that are invisible to the user” (web.dev, updated 4 Sep 2025). Optimize and Adobe both mask with opacity, so while their mask is up there is no LCP candidate at all. “Good” LCP is 2.5 seconds.

Commissioned by Google and conducted by 55 and Deloitte, a study of “37 leading European and American brand sites” across “over 30 million user sessions” found that a 0.1 s speed improvement moved retail shoppers 9.1% more often from product page to add-to-basket, and that those shoppers “spent 9.2% more” (Milliseconds Make Millions, web.dev, 24 Jun 2020, on 2019 data — the study reports progression between funnel steps, not one conversion rate). You are running a test to move conversion a few percent with a mechanism that can cost a second of load time for every visitor, in the test or not.

The library on every page, snippet or no snippet

third-party-web is built from HTTP Archive's Lighthouse crawl of roughly 4 million sites, attributing script execution by domain. Average main-thread time across pages including each vendor, from release 0.29.2 — the current one as of 26 Jul 2026: Kameleoon 1,542 ms · ContentSquare 1,504 ms · Google Tag Manager 1,066 ms · Optimizely 898 ms · Hotjar 732 ms · AB Tasty 524 ms · VWO 465 ms · Microsoft Clarity 405 ms. Two caveats. The README's most recent dataset entry is dated 2024-07-01, so these are the current published figures rather than this month's crawl; and they are correlational, attributed by script origin, and heavy sites run heavy tools. Optimizely does not add 898 ms to your page.

Bytes are more direct. One snapshot, our own measurement, on 26 Jul 2026: curl against the VWO install running on vwo.com's own site pulled a settings payload of 30 KB over the wire that decompressed to 144 KB of JavaScript, plus the async library vanj.js at 38 KB / 131 KB — about 68 KB transferred and 275 KB to parse, before a single experiment did anything. Method, so it reproduces: request j.php with the same a= and u= parameters the live SmartCode sends — u= matters, and dropping it returns a much smaller response. One account, one afternoon; the payload scales with how many experiments an account runs, so treat it as an order of magnitude, not a spec.

Whether it keeps growing depends on the vendor, and two of them publish the answer. AB Tasty: “Some solutions include the entirety of your tests in their script, whether they are suspended or in draft mode. AB Tasty includes only active tests by default” (7 Apr 2017). Optimizely, in Site performance best practices (updated 9 Jul 2026): “The snippet excludes unused audiences and experiments” — but “the basic snippet includes all pages and events you define in a project,” and “to keep the overall snippet size small, avoid creating pages where you do not expect to run experiments.” Ask your vendor which behaviour it has.

The assignment Safari throws away

Two of the biggest client-side vendors document the consequence and recommend the same fix — stop setting the cookie in the browser:

The case for moving the assignment server-side is made by two of the vendors selling client-side, on their own help pages. Be fair about the size of it, though: Optimizely adds a hedge worth quoting alongside the rest — “Re-bucketing instances are likely limited and do not necessarily result in bias or invalidation of experiment results.” VWO implies the opposite. Same mechanism, opposite spin, no clean measurement published by anyone, so treat re-bucketing as noise you can remove rather than a distortion of known size.

How much traffic sits under those rules: Safari is 25.91% of US browser share and Brave — which blocks trackers by default — 2.06%. That is 27.97% of US traffic (StatCounter, all platforms, June 2026).

The event a blocker eats — which server-side doesn't fix

One correction first, because half the pages on this topic are stale. Third-party cookies did not go away in Chrome. Google reversed the deprecation on 22 April 2025 — “we've made the decision to maintain our current approach to offering users third-party cookie choice in Chrome” — and on 17 October 2025 retired most of the Privacy Sandbox APIs, including Topics, Protected Audience and Attribution Reporting, citing “low levels of adoption.” Both on Google's own blog. Anyone selling server-side testing because cookies are disappearing is reading from a 2024 script.

The real problem is Safari plus tracker blocking. Tracker blocking is not ad blocking. EasyList — the ad list — contains zero rules for google-analytics.com, hotjar, optimizely or visualwebsiteoptimizer. EasyPrivacy — the tracker list — blocks clarity.ms, contentsquare.net, crazyegg.com, fullstory.com, google-analytics.com, hotjar.com, kameleoon.com and mouseflow.com by name, plus VWO's server-side tracking endpoint (both lists version 202607252245, downloaded 26 Jul 2026).

Whether that matters depends on which blocker your shoppers run:

Roughly 27 million extension installs, plus Brave's 100M+ monthly actives, block Hotjar, FullStory, Clarity and Google Analytics by name with no configuration — while the two largest extensions in that table don't unless the user turns it on. Blockthrough, who measure this for a living, put typical publisher adblock rates at “10 to 40%” (31 May 2022) and their own measured US rate at 19% (24 Apr 2023). Your store is not a publisher, and searching for this I found no ecommerce-specific measurement of event loss — only publisher adblock rates. Anyone quoting you a single percentage for your store is extrapolating from those.

The number doesn't have to be 40% to be worth fixing. And the fix isn't “go server-side” — it's “stop sending the event to a third-party domain from the browser.” Different decision.

What each side is genuinely better at — and what it costs

In the words of a vendor that sells both — VWO's own comparison (vwo.com/server-side-testing, updated 24 Feb 2026). Row labels and cells verbatim and untruncated; these are four of their six rows, and the two we've left out (Implementation of Tests, Privacy) don't change the picture:

The situation where client-side is simply right: a marketer needs to ship a headline, image, CTA or layout test today, with no engineer and no release train. That is why the category exists — an advantage, not a consolation. The same holds whenever you don't control the response at all: a hosted storefront with no server code leaves no other option. And note the limit on the other side — a visual editor can author a server-side variant, but the variant still has to exist in code before it can be server-rendered. That is the deploy you were trying to avoid.

Six things server-side costs you:

Choose client-side, server-side, or both

Choose client-side if your tests are copy, images, layout and DOM-rendered offers; a marketer owns experimentation and no engineer is assigned to it; your pages already render fast; or you can't deploy server code at all.

Choose server-side if you want to test a price, a bundle rule, a search ranking, a recommendation set or a checkout step; you run paid traffic on decorated URLs where Safari caps the assignment cookie at a day; the mask is measurably hurting your LCP; or the same assignment has to hold across web and a native app.

Choose both if you're running a real programme: marketing runs presentational tests without filing a ticket, engineering runs the backend ones. The trap is going purely server-side — every test then needs engineering time, and the queue becomes the constraint.

Where Mtrix sits, precisely

Mtrix runs both sides. Here is the boundary, including the parts that don't flatter us.

• Allocation is always server-side — every test, every install, computed in Mtrix's backend rather than the browser. And the assignment isn't in the browser either: it's a record in Mtrix's database against your visitor ID, with a 30-day server-side lookback. Nothing there for ITP to wipe.
• The other ITP failure mode stands, and here is the honest version. Mtrix's SDK writes first-party cookies with document.cookie — the same class Safari caps at seven days. On a plain CDN install, a Safari shopper returning after eight idle days is a new visitor to Mtrix too. The integration guide documents the fix Optimizely and VWO also recommend: INTEGRATION.md §4 sets mtrix_user_id from your own server at a one-year maxAge, where the cap doesn't apply. Any vendor claiming its browser script is immune to ITP is either describing a server-set cookie or is wrong.
• Server-side rendering of the variant — the part that removes the flash — needs a developer once. Call getPageDetails() from your server load function and hand the result to the client: about 35 lines, documented for Hydrogen, Next.js and SvelteKit. No dashboard toggle, and not possible on a Shopify Liquid theme — Shopify's own Rollouts swaps whole themes and checkout configurations server-side instead. Called from the browser, a Mtrix test flashes like any other. We'd rather say that than let you find out in week two. On a store built and hosted on Mtrix, none of that applies — there is no load function to write. Until an experiment is created and in progress, the original renders for that section; once the experiment is live, the variant is chosen per the allocation.
• Mtrix is web only. No native iOS or Android SDK; Mtrix records mobile web traffic. If the assignment has to hold inside a native app — the row VWO's table above sells hardest — pick a vendor that ships a mobile SDK, not Mtrix.
• Events go where you point them. By default, a browser request to Mtrix's API. Route them through your own domain — an /api/events route on your server — and nothing is stripped. A wiring decision, not the default install. Mtrix sets no third-party cookies at all: identity travels in the request body.
• 100% of sessions captured, on every install; sampling is an opt-in control, not a plan tier. No cap on experiments either — collision is the real constraint, and audience targeting removes it by bucketing on the server so overlapping tests are mutually exclusive. Don't read that as unique — mutual exclusion is standard in mature experimentation tools. What differs is the packaging: VWO's own pricing page puts mutually exclusive groups on Enterprise only, capped at 10 user groups (read 26 Jul 2026). Mtrix doesn't tier it away, and it applies to element-level tests and behavioural segments, not just whole-page swaps.
• A losing test is a starting point. Filter session replay to the losing arm and watch what those shoppers actually did — same script that ran the test.

What vendor pages leave out

Flicker is partly a heavy-tool problem, not purely a client-side one. A small script placed early in the head flickers far less than a payload the size of the one measured above pulled through a tag manager. If your only issue is a visible flash on presentational tests, a lighter client-side tool fixes it for a fraction of the effort of changing architectures. Vendors selling server-side rarely mention this.

More tests is not better tests. Velocity without traffic produces confident-looking noise, and most DTC stores can power only a handful of well-designed tests at once. Traffic is the binding constraint, not your licence.

FAQ