Stop shipping on opinion. Run a controlled experiment to learn which variation wins with statistical confidence — or hand a multi-armed bandit the wheel and let it shift traffic to the best performer automatically, in real time.
Both start the same way — you have a few versions of something and a metric you want to move. What differs is what you want out of it.
An experiment (A/B/n) is for learning. You split traffic evenly, hold it steady, and measure until the result is statistically trustworthy. The payoff is a confident, explainable decision: "Free Upgrade beat the others on bookings — ship it." You get the winner and the why.
A multi-armed bandit is for earning. Instead of waiting for a verdict, it continuously shifts more traffic toward whatever's winning right now, so you capture more of the reward during the test. Perfect when the "best" can drift and you care more about the outcome than the explanation.
Want a decision you'll ship and explain? Run an experiment. Want to maximize a metric continuously with no manual call? Use a bandit.
Same inputs, different goal: understand vs. optimize
Experimentation turns "we think" into "we know" — and each role gets a different kind of certainty.
| Persona | What they care about | How experiments & bandits deliver |
|---|---|---|
| 👔 C-suite (CEO / CPO) | Revenue impact, defensible bets, not shipping on hunches | Every meaningful change is tied to a measured outcome. Roadmap debates end with evidence; bandits quietly bank the upside on high-velocity surfaces. |
| 🧭 Product managers | Which variation actually converts, and by how much | Launch four promos, let the experiment name the winner on bookings with confidence — then ship it and move on to the next test, no engineer required. |
| 📈 Growth / Marketing | Squeezing conversion out of creative and merchandising | A bandit auto-rotates promos, rankings, or hero copy toward the best performer in real time — capturing more conversions than a fixed test would while it runs. |
| 📊 Data / Analytics | Statistical rigor, trustworthy readouts, no p-hacking | Bayesian or frequentist analysis with credible/confidence intervals, built in. Results are consistent and shareable instead of one-off spreadsheets. |
| 💻 Engineers | Not building (and babysitting) a homegrown test framework | Experiments run on the same flags you already ship behind. No custom bucketing code, no experiment plumbing — instrument a metric and go. |
Two tests run continuously on the 24/7 traffic. The "Promo Banner Messaging" experiment splits the top-of-page promo four ways and measures which drives the most bookings. The "Search Ranking Optimizer" bandit tries four sort orders for search results and auto-shifts toward whichever earns the most destination views. The load generator gently biases the outcome so there's a clear winner to point at: Free Upgrade for the experiment, Trending for the bandit.
The promo-banner experiment: four arms measured on booking conversion
The photo platform (which also runs Flickr) had a homegrown PHP A/B framework that needed custom code for every experiment. With LaunchDarkly, product managers now run A/B tests themselves from a control panel — no developer, no deploy — iterating on trial offers and designs to turn more free users into subscribers. "Whenever someone does something as basic as click a button, the click is tracked, and we can use that information," says front-end engineer Sheryl Lun.
Expand each topic for real configuration and code from this app, with the LD docs to go deeper.
An experiment holds allocation fixed so it can measure each arm fairly. It answers "which is best, and how sure are we?" — then you decide. Use it for changes you'll commit to and want to understand: checkout flows, pricing, onboarding.
A multi-armed bandit treats allocation as the thing to optimize. It balances exploring the arms against exploiting the current leader, reallocating traffic on a cadence so more users get the winning variation while the test runs. Use it where the reward matters more than the explanation and the best option can change: promo creative, search ranking, recommendations.
Both use the same building blocks in LaunchDarkly — a flag, its variations, a metric, and a randomization unit — so moving between them is a config choice, not a rebuild.
The top-of-page promo banner (promo-banner-text) has four variations: No Promo, Flash Sale, Free Upgrade, and Bundle Deal. The experiment splits users evenly (25% each) and measures:
booking-conversion: did the user reach Confirm & Pay? (higher is better)promo-click: did they click the banner's CTA?Allocation stays fixed so each promo gets a fair, comparable sample. Over hours of traffic LaunchDarkly builds a per-arm conversion rate with intervals, and the winner emerges with confidence — not a gut call. In this demo the load generator nudges Free Upgrade users to complete bookings more often, so it pulls ahead as the clear winner.
nav.js fires promo-click when the CTA is clicked; booking.html fires confirm-booking at Confirm & Pay. Both attribute to the promo variation the user saw.
The four promo creatives under test, as rendered on the site
The search-ranking flag drives how destination results are sorted: Recommended, Price: low to high, Price: high to low, or Trending. The bandit's reward is destination-view — a user opening a destination detail page — and it reallocates hourly, moving more traffic to the sort order producing the most views.
Unlike the experiment, there's no waiting for a verdict: the bandit is already capturing the upside as it learns. In this demo the load generator makes Trending shoppers browse more destinations, so the bandit steadily concentrates traffic on Trending.
The flag value flows straight into the query — search.html evaluates search-ranking after LD init (which buckets the user) and passes it to /api/search, where destinationService.search maps it to an ORDER BY.
The bandit reallocating traffic toward the highest-reward sort order
Three custom metrics power the two analyses, all "higher is better," all keyed on the user randomization unit:
booking-conversion (event confirm-booking) — experiment primarypromo-click (event promo-click) — experiment secondarydestination-view (event destination-view) — bandit rewardThe events are fired client-side against the same LD context that evaluated the flag, so LaunchDarkly attributes each one to the right variation:
Both analyses are created idempotently at conductor boot (scripts/ld-experiment-setup.js) through the same stable POST /experiments endpoint. The only real difference is the type field — and the bandit adds a reallocation cadence:
Each defines its treatments (a flag variation + allocation), its metrics, and a randomization unit, then a startIteration instruction sets it running. Most teams do all of this in the LaunchDarkly UI in a couple of minutes; the API path here just lets the demo self-provision.
LaunchDarkly reports each arm with an interval — a Bayesian credible interval by default, or frequentist confidence intervals — so you can see not just which variation leads but how much evidence backs it. The experiment surfaces a probability-to-be-best per arm; the bandit shows allocation drifting toward the leader over time.
Unlike the guarded rollout (which decides in minutes because a 100%-broken treatment is an unmistakable signal), experiments need enough per-arm samples to reach significance. With this app's traffic that's a matter of hours, not seconds — this is a "check it tomorrow" demo, and the load-gen bias makes the eventual winner decisive rather than a coin-flip.
The flag evaluations that bucket users, the metric events that score them, and the session replays that show them are all keyed to one LD context — so an experiment arm links straight to the replays of users who saw it.
Per-arm results with intervals — the winner, and the confidence behind it