JSON A/B Testing: Experiment Config, Assignment API & Event Tracking

Written and reviewed by the Jsonic editorial team — every guide is verified against the official spec or runtime before publication.

Last updated: May 20, 2026

Most A/B testing articles focus on UI tooling and dashboards. This guide covers the JSON API layer: how to design the experiment configuration schema, the assignment API response format, consistent hashing for reproducible bucketing, the event tracking payload, statistical results in JSON, server-side vs. client-side assignment in Next.js, and multi-arm bandit configuration with Thompson Sampling. Every section includes a concrete JSON example and the reasoning behind each field choice.

JSON Experiment Configuration Schema

The experiment configuration is the source of truth for the assignment service, the analytics pipeline, and the feature flag system. It must encode enough information for any service to independently compute a deterministic assignment — including variant weights, targeting conditions, and the per-variant config object that the application reads to know what to render.

// Experiment configuration JSON schema
{
  "experimentId": "checkout-cta-v2",          // unique slug or UUID
  "name": "Checkout CTA Button Copy Test",
  "description": "Tests 'Buy Now' vs 'Complete Purchase' copy on checkout",
  "status": "running",                         // draft | running | paused | completed
  "startDate": "2026-02-15T00:00:00Z",        // ISO 8601
  "endDate": "2026-03-15T00:00:00Z",
  "salt": "a3f9b2c7",                         // random salt for hashing independence
  "minimumSampleSize": 2000,                  // per variant before trusting results

  "variants": [
    {
      "id": "control",
      "name": "Control — Buy Now",
      "weight": 50,                           // integer, all weights must sum to 100
      "config": {
        "buttonText": "Buy Now",
        "buttonColor": "#0070f3"
      }
    },
    {
      "id": "variant-a",
      "name": "Variant A — Complete Purchase",
      "weight": 50,
      "config": {
        "buttonText": "Complete Purchase",
        "buttonColor": "#0070f3"
      }
    }
  ],

  "metrics": [
    {
      "id": "checkout_conversion",
      "name": "Checkout Conversion Rate",
      "type": "binomial",                     // binomial | continuous | ratio
      "isPrimary": true
    },
    {
      "id": "revenue_per_user",
      "name": "Revenue Per User",
      "type": "continuous",
      "isPrimary": false
    }
  ],

  "targeting": {
    "percentTraffic": 100,                    // expose 100% of eligible users
    "conditions": [
      { "attribute": "userCountry", "operator": "in", "values": ["US", "CA", "GB"] },
      { "attribute": "deviceType",  "operator": "eq", "value": "desktop" },
      { "attribute": "accountAgeDays", "operator": "gte", "value": 7 }
    ]
  },

  "mutuallyExclusiveGroup": "checkout-layer", // prevents collision with other checkout experiments
  "createdBy": "alice@example.com",
  "createdAt": "2026-02-10T09:00:00Z"
}

The config object within each variant is intentionally open-ended — it holds whatever parameters differ between variants. The assignment service reads the experiment-level metadata (weights, targeting, salt) and returns the assigned variant's config to the application. The application never needs to know about other variants. The salt field is critical for experiment independence: without a unique salt per experiment, assignments across experiments are correlated. See JSON Schema validation for how to validate this config at write time.

Assignment API Response Format

The assignment API is the runtime interface between the experiment system and the application. A GET request to /experiment/assignreturns a deterministic variant assignment for a given user and experiment. The response must be fast (<5ms p99 via Redis O(1) lookup), cacheable, and include enough context for the application to render the correct experience and for the analytics pipeline to attribute subsequent events.

// GET /experiment/assign?experimentId=checkout-cta-v2&userId=usr_7f3a9c
// Response: 200 OK
{
  "experimentId": "checkout-cta-v2",
  "variantId": "variant-a",
  "userId": "usr_7f3a9c",
  "assignedAt": "2026-02-20T14:32:00.000Z",   // ISO 8601, millisecond precision
  "config": {
    "buttonText": "Complete Purchase",
    "buttonColor": "#0070f3"
  },
  "isControl": false,
  "debug": {                                    // omit in production; include in staging
    "bucket": 73,                              // computed hash bucket 0-99
    "variantRanges": {
      "control": [0, 49],
      "variant-a": [50, 99]
    }
  }
}

// Bulk assignment: POST /experiment/assign (batch for multiple experiments)
// Request body:
{
  "userId": "usr_7f3a9c",
  "experimentIds": ["checkout-cta-v2", "pricing-page-v3", "nav-redesign-v1"]
}
// Response:
{
  "userId": "usr_7f3a9c",
  "assignments": {
    "checkout-cta-v2":  { "variantId": "variant-a", "config": { "buttonText": "Complete Purchase" } },
    "pricing-page-v3":  { "variantId": "control",   "config": { "plan": "standard" } },
    "nav-redesign-v1":  { "variantId": "variant-b",  "config": { "layout": "horizontal" } }
  }
}

// Error response: user does not meet targeting conditions
// HTTP 200 (not 4xx — the request succeeded; the user is simply not in the experiment)
{
  "experimentId": "checkout-cta-v2",
  "variantId": null,
  "userId": "usr_7f3a9c",
  "assignedAt": "2026-02-20T14:32:00.000Z",
  "config": null,
  "excluded": true,
  "excludedReason": "targeting_mismatch"      // targeting_mismatch | experiment_not_running | holdout
}

// Caching strategy:
// - Cache assignment in Redis with key "assign:{experimentId}:{userId}" TTL 24h
// - Set a cookie "exp_checkout-cta-v2=variant-a; Max-Age=2592000; SameSite=Lax"
// - On subsequent requests, read cookie directly — no Redis call needed

Return HTTP 200 even when the user is excluded from the experiment — a 4xx response would cause clients to error-handle rather than render the default experience. The config field is the application's single source of truth for what to render; the variantId is for analytics. Cache assignments aggressively: compute the hash once, store in Redis, and persist in a cookie so the middleware can serve assignments without any downstream call.

Consistent User Assignment for JSON Experiments

Consistent hashing ensures a user always sees the same variant for a given experiment, without requiring a database lookup for every request. The algorithm maps hash(userId + experimentId + salt) % 100 to a bucket (0–99), then assigns the bucket to a variant based on cumulative weight ranges. The salt ensures assignments are statistically independent across experiments — a user in the "control" group for experiment A has an equal probability of being in any variant of experiment B.

// TypeScript: consistent hash assignment (runs at edge, <1ms)
import { createHash } from 'crypto'

interface Variant {
  id: string
  weight: number    // integer, weights must sum to 100
  config: Record<string, unknown>
}

interface Experiment {
  experimentId: string
  salt: string
  variants: Variant[]
}

function assignVariant(
  experiment: Experiment,
  userId: string
): { variantId: string; bucket: number; config: Record<string, unknown> } {
  // 1. Hash: combine userId + experimentId + salt for independence
  const input = userId + experiment.experimentId + experiment.salt
  const hash  = createHash('sha256').update(input).digest('hex')

  // 2. Map hash to bucket 0–99
  // Use first 8 hex chars (32 bits) for sufficient entropy
  const bucket = parseInt(hash.slice(0, 8), 16) % 100

  // 3. Map bucket to variant by cumulative weight
  let cumulative = 0
  for (const variant of experiment.variants) {
    cumulative += variant.weight
    if (bucket < cumulative) {
      return { variantId: variant.id, bucket, config: variant.config }
    }
  }

  // Fallback (should never reach here if weights sum to 100)
  const last = experiment.variants[experiment.variants.length - 1]
  return { variantId: last.id, bucket, config: last.config }
}

// Example usage:
const experiment: Experiment = {
  experimentId: 'checkout-cta-v2',
  salt: 'a3f9b2c7',
  variants: [
    { id: 'control',   weight: 50, config: { buttonText: 'Buy Now' } },
    { id: 'variant-a', weight: 50, config: { buttonText: 'Complete Purchase' } },
  ],
}

// Same userId always produces the same bucket:
assignVariant(experiment, 'usr_7f3a9c')
// => { variantId: 'variant-a', bucket: 73, config: { buttonText: 'Complete Purchase' } }

assignVariant(experiment, 'usr_7f3a9c')
// => { variantId: 'variant-a', bucket: 73, config: { buttonText: 'Complete Purchase' } }

// Different userId produces independent bucket (no correlation with other experiments):
assignVariant(experiment, 'usr_2b8d1e')
// => { variantId: 'control', bucket: 21, config: { buttonText: 'Buy Now' } }

// Verify independence: same user in different experiment (different salt)
const otherExperiment = { ...experiment, experimentId: 'pricing-page-v3', salt: 'd7e4f1a2' }
assignVariant(otherExperiment, 'usr_7f3a9c')
// => Different bucket — salt ensures no correlation

// Sample size calculation for 95% confidence, 80% power:
// n = 2 × (1.96 + 0.84)² × p(1-p) / δ²
// For p=0.05 (5% baseline), δ=0.01 (detect 1pp lift):
// n = 2 × 7.84 × 0.0475 / 0.0001 = 7,448 users per variant
function sampleSizePerVariant(
  baselineRate: number,    // e.g., 0.05 for 5%
  mde: number,             // minimum detectable effect, e.g., 0.01 for 1pp
  alpha = 0.05,            // significance level (two-tailed)
  power = 0.80             // 1 - beta
): number {
  const zAlpha = 1.96  // z-score for alpha=0.05 two-tailed
  const zBeta  = 0.84  // z-score for power=0.80
  const p = baselineRate
  return Math.ceil(2 * Math.pow(zAlpha + zBeta, 2) * p * (1 - p) / Math.pow(mde, 2))
}

The salt is not a secret — it is a random string stored in the experiment config to prevent hash correlation between experiments. Generate a fresh 8-character random hex string for each experiment at creation time. Without the salt, a user with userId = "usr_7f3a9c" would produce the same hash prefix for every experiment with the same ID prefix, causing non-random assignment distribution. MurmurHash3 or xxHash are faster alternatives to SHA-256 for high-throughput edge environments.

Event Tracking JSON Schema for A/B Tests

Event tracking is the data pipeline that connects user behavior back to experiment variants. Every event must carry its experiment context — experimentId and variantId — so your analytics system can compute per-variant metrics. Two event types are special: "exposure" (tracked when the user first sees the variant) and "conversion" (tracked when the primary metric is triggered).

// Single event payload (self-contained — no foreign key lookups needed)
{
  "type": "conversion",                       // exposure | conversion | click | page_view | custom
  "experimentId": "checkout-cta-v2",
  "variantId": "variant-a",
  "userId": "usr_7f3a9c",
  "sessionId": "sess_4a8b2d",               // groups events within a session
  "clientTimestamp": "2026-02-20T14:32:00.123Z",  // client-reported (may drift)
  "serverTimestamp": "2026-02-20T14:32:00.145Z",  // server-recorded on receipt
  "properties": {                            // open-ended event-specific data
    "revenue": 149.99,
    "currency": "USD",
    "productId": "prod_9c3f2a",
    "checkoutStep": "confirmation"
  }
}

// Exposure event — MUST be tracked when user first sees the variant
// (without exposure events, conversion rate denominator is wrong)
{
  "type": "exposure",
  "experimentId": "checkout-cta-v2",
  "variantId": "variant-a",
  "userId": "usr_7f3a9c",
  "sessionId": "sess_4a8b2d",
  "clientTimestamp": "2026-02-20T14:31:55.000Z",
  "serverTimestamp": "2026-02-20T14:31:55.022Z",
  "properties": {
    "page": "/checkout",
    "renderMethod": "ssr"                   // ssr | csr — for debugging hydration issues
  }
}

// Batched event payload (collect 5s or 100 events, then POST array)
// POST /events/batch
{
  "batchId": "bat_f2a9c3",               // UUID for deduplication
  "sentAt": "2026-02-20T14:32:05.000Z",  // when the batch was sent
  "events": [
    {
      "type": "exposure",
      "experimentId": "checkout-cta-v2",
      "variantId": "variant-a",
      "userId": "usr_7f3a9c",
      "sessionId": "sess_4a8b2d",
      "clientTimestamp": "2026-02-20T14:31:55.000Z",
      "properties": { "page": "/checkout" }
    },
    {
      "type": "click",
      "experimentId": "checkout-cta-v2",
      "variantId": "variant-a",
      "userId": "usr_7f3a9c",
      "sessionId": "sess_4a8b2d",
      "clientTimestamp": "2026-02-20T14:32:00.000Z",
      "properties": { "element": "cta-button", "page": "/checkout" }
    },
    {
      "type": "conversion",
      "experimentId": "checkout-cta-v2",
      "variantId": "variant-a",
      "userId": "usr_7f3a9c",
      "sessionId": "sess_4a8b2d",
      "clientTimestamp": "2026-02-20T14:32:00.123Z",
      "properties": { "revenue": 149.99, "currency": "USD" }
    }
  ]
}

// Client-side batching logic (TypeScript):
class EventBatcher {
  private buffer: Event[] = []
  private timer: ReturnType<typeof setTimeout> | null = null
  private readonly maxSize = 100
  private readonly maxDelayMs = 5000

  track(event: Event): void {
    this.buffer.push(event)
    if (this.buffer.length >= this.maxSize) {
      this.flush()
    } else if (!this.timer) {
      this.timer = setTimeout(() => this.flush(), this.maxDelayMs)
    }
  }

  private async flush(): Promise<void> {
    if (this.buffer.length === 0) return
    const batch = this.buffer.splice(0)        // drain buffer atomically
    if (this.timer) { clearTimeout(this.timer); this.timer = null }
    await fetch('/events/batch', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ batchId: crypto.randomUUID(), sentAt: new Date().toISOString(), events: batch }),
      keepalive: true,                         // survives page unload
    })
  }
}

The keepalive: true flag on the fetch call is essential — without it, events tracked just before page navigation (e.g., a click that triggers a redirect) are dropped when the browser terminates the request. The batchId enables server-side deduplication: if the client retries a failed POST, the server can discard the duplicate batch. Track exposure events as early as possible — before any API call or async rendering — to maximize denominator accuracy.

Statistical Results in JSON Format

The results API returns the computed statistics for each variant so dashboards and alerting systems can consume experiment outcomes programmatically. The JSON structure separates experiment metadata, per-variant metrics, and the statistical comparison between control and treatment.

// GET /experiment/results?experimentId=checkout-cta-v2
{
  "experimentId": "checkout-cta-v2",
  "name": "Checkout CTA Button Copy Test",
  "status": "running",
  "startDate": "2026-02-15T00:00:00Z",
  "primaryMetric": "checkout_conversion",
  "computedAt": "2026-02-20T00:00:00Z",     // when statistics were last computed
  "totalUsers": 8432,                        // unique users exposed across all variants
  "winner": null,                            // null while running; variantId when concluded

  "variants": [
    {
      "variantId": "control",
      "isControl": true,
      "sampleSize": 4218,                    // unique users exposed to this variant
      "conversions": 211,                    // users who completed the primary metric
      "conversionRate": 0.0500,              // conversions / sampleSize
      "statistics": {
        "confidenceInterval": [0.0435, 0.0570],  // 95% Wilson score interval [lower, upper]
        "standardError": 0.00335,
        "lift": null,                             // undefined for control
        "pValue": null,                           // undefined for control
        "isSignificant": null
      }
    },
    {
      "variantId": "variant-a",
      "isControl": false,
      "sampleSize": 4214,
      "conversions": 253,
      "conversionRate": 0.0600,
      "statistics": {
        "confidenceInterval": [0.0530, 0.0676],  // 95% Wilson score interval
        "standardError": 0.00366,
        "lift": 0.200,                            // (0.06 - 0.05) / 0.05 = 20% relative lift
        "liftAbsolute": 0.010,                    // 0.06 - 0.05 = 1pp absolute lift
        "pValue": 0.018,                          // two-tailed z-test
        "zScore": 2.37,
        "isSignificant": true                     // pValue < 0.05
      }
    }
  ],

  "sampleSizeRequired": 7448,               // per variant for 80% power at 1pp MDE
  "sampleSizeReached": true,               // all variants have >= minimumSampleSize

  "notes": "Variant A shows 20% relative lift in conversion rate with p=0.018. Recommend shipping after 7 more days to validate stability."
}

// Computing Wilson score confidence interval (more accurate than normal approx at low n):
function wilsonInterval(
  successes: number,
  n: number,
  z = 1.96       // 1.96 for 95% confidence
): [number, number] {
  const p     = successes / n
  const denom = 1 + z * z / n
  const center = (p + z * z / (2 * n)) / denom
  const margin = (z * Math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
  return [center - margin, center + margin]
}

// Two-proportion z-test p-value:
function twoProportionPValue(
  n1: number, c1: number,   // control: n, conversions
  n2: number, c2: number    // treatment: n, conversions
): number {
  const p1     = c1 / n1
  const p2     = c2 / n2
  const pPool  = (c1 + c2) / (n1 + n2)
  const se     = Math.sqrt(pPool * (1 - pPool) * (1 / n1 + 1 / n2))
  const z      = (p2 - p1) / se
  // Two-tailed p-value (approximation via standard normal CDF):
  return 2 * (1 - standardNormalCdf(Math.abs(z)))
}

Use Wilson score confidence intervals rather than normal approximation — the normal approximation produces intervals outside [0, 1] at low sample sizes or extreme conversion rates. Report both absolute lift (liftAbsolute) and relative lift (lift) — a 20% relative lift sounds impressive but is only 1 percentage point if the baseline is 5%. A statistically significant result with a tiny absolute effect may not be worth the engineering cost to ship. See JSON API design for response envelope patterns.

Server-Side vs Client-Side JSON Assignment in Next.js

In a Next.js App Router application, assignment should happen server-side in middleware before the page renders. This eliminates variant flicker (the flash of the control variant before JavaScript loads), works with SSR and streaming, and avoids React hydration mismatches from rendering different content on the server vs. client.

// middleware.ts — assign variants at the edge before page renders
import { NextRequest, NextResponse } from 'next/server'
import { assignVariant } from '@/lib/experiments/assign'
import { getExperiments }  from '@/lib/experiments/registry'

export async function middleware(request: NextRequest) {
  const response = NextResponse.next()

  // 1. Get userId from session cookie (or anonymous ID)
  const userId = request.cookies.get('userId')?.value ?? generateAnonymousId()

  // 2. Load active experiments for this route
  const experiments = getExperiments(request.nextUrl.pathname)

  for (const experiment of experiments) {
    const cookieName = `exp_${experiment.experimentId}`

    // 3. Check if assignment is already cached in a cookie
    if (!request.cookies.has(cookieName)) {
      // 4. Compute deterministic assignment (no network call — pure hash)
      const assignment = assignVariant(experiment, userId)

      // 5. Set cookie so subsequent requests skip computation
      response.cookies.set(cookieName, JSON.stringify({
        variantId: assignment.variantId,
        config:    assignment.config,
      }), {
        maxAge:   60 * 60 * 24 * 30,   // 30 days
        sameSite: 'lax',
        httpOnly: false,               // client needs to read it for analytics events
      })
    }
  }

  return response
}

export const config = {
  matcher: ['/checkout/:path*', '/pricing/:path*'],  // only run on relevant routes
}

// app/checkout/page.tsx — Server Component reads assignment from cookie
import { cookies } from 'next/headers'

export default async function CheckoutPage() {
  const cookieStore = cookies()
  const raw = cookieStore.get('exp_checkout-cta-v2')?.value

  // Parse assignment from cookie — set by middleware
  const assignment = raw ? JSON.parse(raw) : { variantId: 'control', config: { buttonText: 'Buy Now' } }

  return (
    <CheckoutForm
      variantId={assignment.variantId}
      buttonText={assignment.config.buttonText}
    />
  )
}

// app/checkout/checkout-form.tsx — Client Component tracks exposure
'use client'
import { useEffect } from 'react'
import { trackEvent } from '@/lib/analytics'

interface Props {
  variantId: string
  buttonText: string
}

export function CheckoutForm({ variantId, buttonText }: Props) {
  useEffect(() => {
    // Track exposure once on mount — variantId comes from server, no flicker
    trackEvent({
      type: 'exposure',
      experimentId: 'checkout-cta-v2',
      variantId,
      properties: { page: '/checkout', renderMethod: 'ssr' },
    })
  }, [variantId])

  return (
    <form>
      {/* form fields */}
      <button type="submit">{buttonText}</button>
    </form>
  )
}

// ANTI-PATTERN: reading assignment in useEffect causes flicker and hydration mismatch
// ❌ DO NOT DO THIS:
// useEffect(() => {
//   const cookie = document.cookie.match(/exp_checkout-cta-v2=([^;]+)/)?.[1]
//   const { variantId } = JSON.parse(decodeURIComponent(cookie ?? '{}'))
//   setButtonText(variantId === 'variant-a' ? 'Complete Purchase' : 'Buy Now')
// }, [])
// This renders 'Buy Now' on server, then switches to 'Complete Purchase' after hydration — visible flicker

The middleware runs at the edge (Vercel Edge Runtime) before any server component code executes, so the assignment is available immediately when the Server Component reads cookies(). The assignment cookie is not httpOnly because the client-side analytics SDK needs to read it to include variantId in event payloads without an additional API call. Never derive the rendered variant from a useEffect — this guarantees flicker and hydration errors.

Multi-Arm Bandit JSON Configuration

A standard A/B test uses fixed weights (e.g., 50/50) for the entire experiment duration, wasting traffic on poor-performing variants. A multi-arm bandit dynamically adjusts weights based on observed performance. Thompson Sampling, the most common algorithm, maintains a Beta distribution per variant and assigns each user to the variant with the highest sampled value — naturally routing more traffic to better performers while continuing to explore others.

// Multi-arm bandit experiment configuration
{
  "experimentId": "homepage-hero-bandit",
  "name": "Homepage Hero Image Bandit",
  "algorithm": "thompson_sampling",           // "fixed" for standard A/B, "thompson_sampling" for bandit
  "status": "running",
  "startDate": "2026-02-15T00:00:00Z",
  "updatedAt": "2026-02-20T06:00:00Z",       // last time weights were recomputed

  "variants": [
    {
      "id": "hero-product",
      "name": "Product Hero Image",
      "weight": 23,                           // DYNAMIC — recomputed every hour from beta params
      "config": { "heroImage": "product.jpg", "headline": "Build faster" },
      "betaParams": {
        "alpha": 48,                          // 1 + number of successes (conversions)
        "beta":  156                          // 1 + number of failures (non-conversions)
        // Posterior mean = alpha / (alpha + beta) = 48/204 = 23.5% conversion rate estimate
      }
    },
    {
      "id": "hero-social-proof",
      "name": "Social Proof Hero",
      "weight": 54,                           // higher weight — currently best performer
      "config": { "heroImage": "team.jpg", "headline": "Trusted by 10,000 teams" },
      "betaParams": {
        "alpha": 112,
        "beta":  187
        // Posterior mean = 112/299 = 37.5% — best performer, gets most traffic
      }
    },
    {
      "id": "hero-demo",
      "name": "Demo CTA Hero",
      "weight": 23,
      "config": { "heroImage": "demo.jpg", "headline": "See it in action" },
      "betaParams": {
        "alpha": 29,
        "beta":  94
        // Posterior mean = 29/123 = 23.6%
      }
    }
  ],

  "updateFrequency": "1h",                  // how often to recompute weights from beta params
  "minimumExplorationRate": 0.05,           // always allocate at least 5% to each variant
  "metrics": [{ "id": "signup_click", "type": "binomial", "isPrimary": true }]
}

// Thompson Sampling: weight computation from beta params
// Run this every hour; update experiment config with new weights
function computeThompsonWeights(
  variants: Array<{ id: string; betaParams: { alpha: number; beta: number } }>,
  simulations = 10_000,
  minExploration = 0.05
): Record<string, number> {
  // Count how many simulations each variant wins
  const wins: Record<string, number> = Object.fromEntries(variants.map(v => [v.id, 0]))

  for (let i = 0; i < simulations; i++) {
    // Sample from each variant's Beta(alpha, beta) distribution
    const samples = variants.map(v => ({
      id:     v.id,
      sample: sampleBeta(v.betaParams.alpha, v.betaParams.beta),
    }))
    // Variant with highest sample wins this simulation
    const winner = samples.reduce((best, cur) => cur.sample > best.sample ? cur : best)
    wins[winner.id]++
  }

  // Convert win counts to percentages, apply minimum exploration floor
  const rawWeights: Record<string, number> = {}
  for (const [id, count] of Object.entries(wins)) {
    rawWeights[id] = count / simulations
  }

  // Apply minimum exploration: ensure each variant gets at least minExploration traffic
  const n = variants.length
  const floor = minExploration
  const adjusted: Record<string, number> = {}
  const totalFloor = floor * n

  for (const [id, w] of Object.entries(rawWeights)) {
    adjusted[id] = Math.max(floor, w * (1 - totalFloor) + floor)
  }

  // Normalize to sum to 1, convert to integer percentages summing to 100
  return normalizeToIntegerWeights(adjusted)
}

// After each conversion or non-conversion, update beta params:
function recordOutcome(variant: { betaParams: { alpha: number; beta: number } }, converted: boolean) {
  if (converted) {
    variant.betaParams.alpha += 1   // success: increment alpha
  } else {
    variant.betaParams.beta  += 1   // failure: increment beta
  }
  // Persist updated params back to experiment config store
}

// Bandit reduces regret by 20-40% vs fixed A/B for short experiments:
// - At 1,000 total users: bandit routes ~70% traffic to winning variant automatically
// - A/B test at 1,000 users: still 50/50, needs more data before switching
// Trade-off: bandit provides less statistical precision for losing variants
// Use A/B for: slow metrics (retention, LTV), regulatory compliance, learning
// Use bandit for: fast binary metrics (click, signup), revenue optimization, short experiments

The betaParams in the JSON config are the persistent state of the bandit — they accumulate across all users and are updated after every conversion event. The weight field is recomputed hourly from the beta parameters via Thompson Sampling simulation. The minimumExplorationRate prevents the bandit from fully converging on one variant too early, ensuring continued data collection on other variants in case conditions change. For long-running experiments measuring retention or LTV, use a standard A/B test — bandits optimize for fast signals and can prematurely converge on a variant that looks good short-term.

Key Terms

A/B Test

A controlled experiment that randomly assigns users to one of two or more variants of a feature and measures the effect on one or more metrics. The control group sees the existing experience; treatment groups see the experimental change. Users are assigned via a deterministic hash of their user ID so the assignment is stable across sessions. The experiment runs until the required sample size is reached, then statistical tests determine whether observed differences are due to the change or random chance. A/B tests require careful design: defining a primary metric before the experiment starts, computing the required sample size to detect the minimum effect of interest, and avoiding peeking at results early (which inflates false positive rates). In JSON API systems, A/B tests manifest as configuration schemas, assignment API responses, and event tracking payloads.

Variant

One of the possible experiences in an A/B test. Each variant has an ID, a weight (integer representing the percentage of eligible users assigned to it), and a config object containing the parameters that differ from other variants. The control variant is the existing, unchanged experience — it is the baseline against which treatment variants are compared. Variants should differ in exactly one dimension at a time to enable causal attribution; testing multiple changes simultaneously is a multivariate test (MVT) with a different statistical model. In the JSON configuration schema, each variant's config object is the application's source of truth for what to render — the application reads the config and renders accordingly without needing to know which experiment it belongs to.

Assignment API

A service endpoint (typically GET /experiment/assign) that accepts a userId and experimentId and returns a deterministic variant assignment. The assignment API must be fast — response time budget is under 5ms at p99, achieved via Redis O(1) lookups for cached assignments. The response includes the variantId, the variant config object (so the application knows what to render), and metadata like assignedAt for analytics. Assignments are cached in Redis and persisted in cookies so the assignment is not recomputed on every request. The assignment API also handles exclusion cases — returning a null variantId with an excludedReason when a user does not meet targeting conditions, rather than returning an HTTP 4xx error.

Consistent Hashing

A technique for assigning users to experiment variants deterministically without a database lookup. The assignment is computed as bucket = hash(userId + experimentId + salt) % 100, where the salt is a per-experiment random string that ensures assignments are independent across experiments. Because the hash function is deterministic, the same userId always produces the same bucket for a given experiment — the user always sees the same variant, even after clearing cookies or switching devices (as long as the userId persists). The salt prevents correlation: without it, users consistently placed in the control group of experiment A would tend to land in the control group of all other experiments, biasing multi-experiment analyses.

Statistical Significance

A measure of whether an observed difference between variants is likely to be a real effect rather than random sampling variation. Conventionally, a result is statistically significant when the p-value is below 0.05 — meaning there is less than a 5% probability of observing a difference this large by chance if the null hypothesis (no effect) is true. Statistical significance does not measure practical importance: a 0.1% absolute lift can be statistically significant with a large enough sample. In the A/B test results JSON, significance is represented by pValue, zScore, and isSignificant (boolean). Reaching statistical significance requires a pre-determined minimum sample size — peeking at results before that threshold inflates false positive rates (the "peeking problem").

Confidence Interval

A range of plausible values for the true conversion rate (or other metric) of a variant, given the observed data. A 95% confidence interval means that if the experiment were repeated many times, 95% of the computed intervals would contain the true population value. In the results JSON, the confidenceInterval field is a [lower, upper] array for the variant's conversion rate. Use Wilson score intervals rather than normal approximation — the normal approximation produces intervals outside [0, 1] at small sample sizes or extreme rates. The overlap between confidence intervals of two variants is a rough visual indicator of statistical significance, but the proper test is the two-proportion z-test that produces the pValue field.

Multi-Arm Bandit

An online learning algorithm for A/B testing that dynamically adjusts variant weights based on observed performance, routing more traffic to better-performing variants while continuing to explore others. Unlike a standard A/B test with fixed 50/50 weights, a bandit adapts in real time. The name comes from the "multi-armed bandit" problem: a gambler choosing between slot machines (arms) with unknown payout rates, seeking to maximize total reward. Bandits reduce regret — the cumulative reward loss from pulling suboptimal arms — compared to pure A/B tests, especially for short experiments or those with a clear early winner. In the JSON configuration, bandits store per-variant beta distribution parameters (alpha and beta) that accumulate with each success and failure, and recompute variant weights periodically via Thompson Sampling simulation.

Thompson Sampling

A Bayesian algorithm for multi-arm bandit problems. For each variant, it maintains a Beta(alpha, beta) distribution where alpha = 1 + successes and beta = 1 + failures. To assign a user, the algorithm samples one value from each variant's Beta distribution and routes the user to the variant with the highest sample. Variants with more successes have higher-mean distributions and thus tend to get higher samples — but there is always some probability of sampling a lesser variant, providing exploration. As data accumulates, the distributions narrow and the algorithm converges on the best variant. The JSON representation stores alpha and beta per variant; a background job recomputes variant weights hourly by running 10,000 simulations and counting which variant wins each simulation.

Minimum Detectable Effect

The smallest treatment effect (lift) that an experiment is designed to detect with a specified statistical power (typically 80%) and significance level (typically 5%). The MDE determines the required sample size: smaller effects require exponentially larger samples. The formula is n = 2 × (zα + zβ)² × p(1−p) / δ² per variant, where p is the baseline conversion rate and δ is the MDE in absolute percentage points. For example, detecting a 1 percentage point lift on a 5% baseline at 95% confidence requires approximately 7,448 users per variant. Choosing the MDE before starting the experiment is critical — an experiment cannot retrospectively claim to detect an effect smaller than the pre-registered MDE without inflating the false positive rate.

FAQ