Skip to main content

A/B Test Variants

A/B testing lets you run two or more variants of the same campaign or landing page, split traffic between them, and let the platform decide which variant performs best. Assignment is deterministic — the same member always sees the same variant for a given test, so the experiment is consistent across sessions.

When to use it​

  • You're not sure which headline, hero image, or call-to-action will convert best.
  • You want to compare a new mechanic (e.g. spin-the-wheel vs scratch card) against your current default.
  • You want a baseline number for "incrementality" — how much extra conversion the new version brings.

How it works​

You create an ABTest for a campaign and define two or more variants. Each variant has a name, a flag for whether it's the control, and a traffic percentage (an integer 0–100 — they should sum to 100). When a member opens the campaign, the platform hashes (memberId, testId) modulo 100 and assigns them to a variant based on cumulative traffic percentages.

The same member always lands on the same variant for the same test — refreshing, switching devices, or visiting weeks later all yield the same assignment.

The platform tracks:

  • Impressions — how many members were assigned to each variant.
  • Conversions — how many of those completed the campaign goal.
  • Conversion rate — conversions / impressions * 100, rounded to two decimals.
  • Lift over control — for non-control variants, the relative gain vs the control's rate.

Once total impressions reach the minimum sample size, the platform marks the test as having statistical significance and picks the variant with the highest conversion rate as the winner.

Step-by-step​

Create a test​

  1. Go to Campaigns → A/B tests and click Create test.
  2. Set the campaign the test applies to.
  3. Set the minimum sample size — the impression count below which the platform won't declare a winner.
  4. Add at least two variants:
    • Name (free text, displayed to operators)
    • Is control — check exactly one variant as control
    • Traffic percentage — integer 0–100, all variants summing to 100
  5. Save. The test is in Draft.

Run the test​

  1. Open the test and set Status to Running. Only running tests assign members.
  2. The platform will start recording impressions and conversions as members hit the campaign.

Read the results​

The test detail page shows a table of variants with impressions, conversions, conversion rate, and lift vs control. Once the test passes the minimum sample size, the winning variant is highlighted.

You can stop the test at any time by switching its Status to Stopped. Existing assignments are preserved (members keep seeing the variant they were assigned to until the underlying campaign ends).

Limits and gotchas​

  • Traffic percentages should sum to 100. If they don't, the last variant absorbs any overflow — which is rarely what you want.
  • Statistical significance is a sample-size gate, not a real significance test. The platform reports the higher-converting variant as the winner once you cross the minimum sample size. If you need a chi-square test or confidence interval, compute it externally from the impression/conversion counts.
  • Only one A/B test per campaign should be running at a time. Multiple running tests on the same campaign produce mixed traffic and unclear attribution.
  • Members are assigned, not browsers. Anonymous visitors (no member id yet) don't enter the test until they identify themselves. Plan around that if your campaign starts pre-registration.
  • Permission: Pages.Gamification.ABTests.