A/B testing in Rails outside of controllers

The split gem is a lovely little thing for A/B testing in Rails. But its README only shows you one way to use it: from a controller.

class ApplicationController < ActionController::Base
  include Split::Helper
end

class HomeController < ApplicationController
  def index
    @button = ab_test("new_checkout_button", "old", "new")
  end
end

# later, when the user converts
class OrdersController < ApplicationController
  def create
    # ...
    finished("new_checkout_button")
  end
end

That’s fine until your app stops doing everything in controllers. Mine does plenty in GraphQL resolvers, in service objects, and in background jobs - and the conversion I actually care about (“order was paid”) happens deep inside a mutation, not inside a controller code. None of that is documented. So here’s how to do it anyway.

How Split actually decides things

The trick to using Split anywhere is understanding what ab_test needs. It needs a context object - something it can call session, current_user, cookies, request and response on. That’s because Split has to remember which variant a user was assigned, and the persistence is keyed off that object.

In a controller, the context is just self. Here’s a typical setup - Redis for logged-in users keyed by their id, a cookie for everyone else:

# config/initializers/split.rb
Split.configure do |config|
  Split.redis = Redis.new(url: ENV["REDIS_URL"])

  redis_adapter = Split::Persistence::RedisAdapter.with_config(
    lookup_by: ->(context) { context.send(:current_user)&.id },
    expire_seconds: 1.month.to_i
  )

  config.persistence = Split::Persistence::DualAdapter.with_config(
    logged_in: ->(context) { !context.send(:current_user).try(:id).nil? },
    logged_in_adapter: redis_adapter,
    logged_out_adapter: Split::Persistence::CookieAdapter,
    fallback_to_logged_out_adapter: true
  )
end

Look at that lookup_by lambda - context.send(:current_user)&.id. That is why the context needs current_user. Once you internalize this, everything else is obvious: outside a controller, you just hand Split a context that quacks the right way.

GraphQL: wrap Split::Helper in a context object

The idea: a class that includes Split::Helper, re-exposes ab_test as public (Split’s helper methods are private by design), and answers the API a controller would answer.

class ABContext
  include Split::Helper
  public :ab_test, :ab_finished, :ab_user

  def initialize(current_user:, session:)
    @current_user = current_user
    @session = session
  end

  attr_reader :current_user, :session
end

In practice you want the real session, cookies, request and response too, so the cookie adapter and ab_record_extra_info work. Easiest is to build it straight off the controller:

# app/models/a_b_context.rb
class ABContext
  include Split::Helper
  include OrderConversionRecorder

  attr_accessor :request, :session, :current_user, :response, :cookies

  # Split::Helper's methods are private by default - re-expose them
  public :ab_finished, :ab_user, :ab_test

  def initialize(controller)
    self.request = controller.send(:request)
    self.session = controller.send(:session)
    self.current_user = controller.send(:current_user)
    self.response = controller.send(:response)
    self.cookies = controller.send(:cookies)
  end
end

Then stuff it into your GraphQL context:

# app/controllers/graphql_controller.rb
def execute
  context = {
    user: current_user,
    session: session,
    ab_context: ABContext.new(self),
    # ...
  }
  result = MySchema.execute(query, context: context, variables: variables)
  render json: result
end

And use it from a resolver exactly like you’d use ab_test in a controller:

# app/graphql/types/query_type.rb
field :ab_test, String, null: true do
  argument :name, String, required: true
end

def ab_test(name:)
  return unless name == "new_checkout_button"

  context.ab_context.ab_test(name.to_sym, "old", "new")
end

The one gotcha is the public :ab_finished, :ab_user, :ab_test line - without it those methods are private (they come from a mixin meant to be used as self) and you can’t call them through the wrapper.

Recording conversions where the conversion actually happens

Here’s the part Split really doesn’t help you with. A conversion isn’t “the user hit this controller action” - it’s “the user did the thing”, and “the thing” usually happens several layers deep in a service object or a mutation. So the finished call has to travel down there with it.

The gist:

def record_order_conversion(order)
  return unless user_enrolled_in_experiment?("new_checkout_button")

  ab_record_extra_info("new_checkout_button", "amount_cents", order.total.cents)
  ab_record_extra_info("new_checkout_button", "orders_count", 1)
  ab_finished(:new_checkout_button, reset: true)
end

Here’s how it would look like as a module mixed into ABContext (note it’s already included up above):

module OrderConversionRecorder
  def record_order_conversion(order)
    experiment_name = "new_checkout_button"
    return unless user_enrolled_in_experiment?(experiment_name)

    current_version = ab_test(experiment_name.to_sym, "old", "new")

    ab_record_extra_info(experiment_name, "amount_cents", order.total.cents)
    ab_record_extra_info(experiment_name, "items_count", order.items.count)
    ab_record_extra_info(experiment_name, "orders_count", 1)
    order.update(ab_variant: current_version)

    ab_finished(experiment_name.to_sym, reset: true)
  end

  def user_enrolled_in_experiment?(experiment_name)
    experiments = Split::Metric.possible_experiments(experiment_name)
    return false if experiments.blank?

    experiments.each do |experiment|
      return true unless ab_user[experiment.key].nil?
    end

    false
  end
end

Notes:

• user_enrolled_in_experiment? - before recording anything, check ab_user[experiment.key]. If the user was never assigned a variant, calling ab_test here would enroll them at conversion time, which poisons your data. Split::Metric.possible_experiments resolves the metric name to the actual experiments behind it.
• ab_record_extra_info lets you hang arbitrary numbers off the experiment - order value, item count, whatever you want to compare between variants. (This one’s barely in the README either.)

And from the mutation:

# app/graphql/mutations/order_create.rb
def resolve(**args)
  order = OrderCreator.call(args)
  context.ab_context.record_order_conversion(order)
  { order: order }
end

Background jobs: there is no controller

Now it gets worse. Inside a Sidekiq worker or an ActiveJob there’s no request, no session, no cookies - so the “wrap a controller-like object” trick from above falls apart. There’s nothing to wrap.

We have to drop down a level and talk to Split’s own primitives. The skeleton:

redis_adapter = Split::Persistence::RedisAdapter.with_config(
  lookup_by: ->(_context) { user.id }
)
ab_user = Split::User.new(nil, redis_adapter.new(nil))
experiment = Split::ExperimentCatalog.find_or_create(:new_checkout_button, "old", "new")
variant = Split::Trial.new(user: ab_user, experiment: experiment).choose!

Wrapped up into a tiny utility:

class AbTestUtils
  def self.ab_test(user:, experiment_name:, alternatives:)
    experiment_name = experiment_name.to_sym
    redis_adapter = Split::Persistence::RedisAdapter.with_config(
      lookup_by: ->(_context) { user.id },
      expire_seconds: 1.month.to_i
    )

    ab_user = Split::User.new(nil, redis_adapter.new(nil))
    experiment = Split::ExperimentCatalog.find_or_create(experiment_name, *alternatives)
    trial = Split::Trial.new(user: ab_user, experiment: experiment)
    result = trial.choose!
    result.name
  end

  def self.ab_finished(user:, experiment_name:, alternatives:)
    experiment_name = experiment_name.to_sym
    redis_adapter = Split::Persistence::RedisAdapter.with_config(
      lookup_by: ->(_context) { user.id },
      expire_seconds: 1.month.to_i
    )

    ab_user = Split::User.new(nil, redis_adapter.new(nil))
    experiment = Split::ExperimentCatalog.find_or_create(experiment_name, *alternatives)

    # don't enroll a brand-new user from a background job
    return if ab_user[experiment.key].nil?

    trial = Split::Trial.new(user: ab_user, experiment: experiment)
    trial.choose!
    trial.complete!
  end
end

Used from a job:

class ProcessOrderJob < ApplicationJob
  def perform(order_id)
    order = Order.find(order_id)
    # ...
    AbTestUtils.ab_finished(
      user: order.user,
      experiment_name: :new_checkout_button,
      alternatives: ["old", "new"]
    )
  end
end

This isn’t a hack so much as it’s Split with its pants down. Split::Helper#ab_test is Split::Trial#choose!, and #finished is #complete! - the helper just builds the Split::User for you out of the controller context. Here we build it ourselves.

A very importat bit: the lookup_by lambda has to produce the same key as your controller’s Redis adapter. If the web flow keys by current_user.id and your job keys by anything else, the job is looking at a different user, and the conversion won’t match with the assignment.

Wrap-up

Two escape hatches, then:

1. You have a current_user and something session-like (GraphQL, a controller-adjacent service) - wrap Split::Helper in a small object that has the same API as a controller.
2. You don’t have session at all (background jobs) - call Split::User / Split::ExperimentCatalog / Split::Trial directly.