We ship frequently at Fullscript, with most features behind flags managed with Flipper, a Ruby feature-flag library. Developers enable a feature for internal users first, then raise its rollout percentage in stages until all practitioners have access.
Smart Rollouts is a release validation tool that adds a controlled experiment to the feature flag workflow. It divides the actors inside a limited rollout into equal experiment and control groups, keeps assignments stable as enrolment grows, and records exposure in Mixpanel, a product analytics tool, when an actor reaches the feature code. In Flipper, an actor is a user or another record evaluated against a flag.
We shipped Smart Rollouts in Q3 2025. As of July 2026, 11 teams have run 70 Smart Rollouts. The implementation depends on two mechanisms: a second, independently salted hash for stable group assignment and exposure tracking that waits for first use.
Custom experiments before Smart Rollouts
A percentage-based rollout limits how many users a potential bug can reach. Measuring behavioural changes also requires a comparable control group and an exposure record for both groups. Flipper stores gate state (the rules deciding who sees a feature) in the application database, so Mixpanel has no cohort or exposure data unless the application sends it.
Before Smart Rollouts, teams built assignment and tracking logic inside each feature. Frontend, backend, and mobile features had separate implementations. Writing custom code for each experiment risks introducing bugs that invalidate the data. If a bug corrupts tracking, the team can't reconstruct the missed events. They have to fix the code and restart the experiment.
Our experiment records show about 20 custom experiments per quarter in the first half of 2025. At that volume, a hosted experimentation platform didn't justify the integration cost. We chose to extend the Flipper and Mixpanel paths already used by application code, accepting a narrower system limited to binary tests and fixed group sizes.
Design goals and scope
Five goals shaped the implementation:
- Existing flag checks remain unchanged. Developers manage flags in an internal admin drawer, and application code checks them through
FeatureFlag[:name].enabled?(actor). Starting a rollout requires no new experiment code at call sites. - Group assignments remain stable. Raising enrolment from 10% to 40% adds actors without moving existing assignments between experiment and control.
- Assignment and exposure occur at different times. Bulk flag evaluation can create an assignment, while analytics waits until feature code uses the result.
- Evaluation fails closed. An assignment error is logged and returns `false`, leaving unreleased code off.
- Repeated checks avoid database reads. Request-level and Redis caches store rollout metadata and assignments.
Smart Rollouts supports binary on/off features with a fixed 50/50 comparison. Teams use custom code for tests with 3 or more variants. Statistical analysis lives in Mixpanel, and developers manually complete each rollout as Success (100% enabled) or Issues (0% enabled).
The dual-hash split
Starting a Smart Rollout at 10% creates three populations:
Smart Rollout Distribution
The configured percentage covers both comparison groups. The fixed allocation maps approximately half of enrolled actor identifiers to each group, producing cohorts expected to be similar in size.
Flipper calculates rollout membership by hashing the actor's identifier with the feature name. Smart Rollouts splits enrolled actors with another Flipper calculation, using a virtual feature key (#{feature_name}_rollout) in an isolated Flipper::Adapters::Memory instance. A simplified version of the group calculation is:
1def self.calculate_group(subject, feature_name)2 flipper = Flipper.new(Flipper::Adapters::Memory.new)3 feature = flipper[:"#{feature_name}_rollout"]4 feature.enable(Flipper::Types::PercentageOfActors.new(50))56 actor = Flipper::Actor.new(subject.flipper_id)7 feature.enabled?(actor) ? "EXPERIMENT" : "CONTROL"8end
Flipper includes the feature key in its hash input, so the virtual key supplies a separate salt. The isolated memory instance keeps the real flag's gates out of group assignment. Both hashes are deterministic, which preserves existing assignments when enrolment increases.
We could have assigned groups randomly and saved the result to the database on first use. But random assignment creates race conditions. If two requests for the same user happen at the same time, they might calculate different groups. The database would then have to reject one. Because the second hash is deterministic, every request calculates the exact same group before touching the database. A unique index still handles duplicate inserts.
Smart Rollouts stores lifecycle metadata separately from per-actor state. A FlipperExperiment row is the rollout record, with its name, status, and dates. An Experiment row is an assignment, with one actor's group and a tracked_at timestamp. The assignment remains necessary because the hash can't record exposure, and request-level and Redis caches make later reads cheap.
Exposure tracking at first use
The first implementation sent a Mixpanel event when it created an assignment. Fullscript's GraphQL layer evaluates flags in bulk during page load, so that approach could record exposure for users who never reached the feature. Including those users in both groups dilutes any behavioural difference caused by the feature.
Assignment now occurs during bulk evaluation. Analytics waits until feature code reads the flag:
Smart Rollouts lazy tracking diagram
The initial GraphQL response includes enabled flags and keys for untracked assignments. When a React component calls useFlippers for one of those keys, the hook sends a tracking mutation. The server emits a Mixpanel $experiment_started event with the rollout name and the actor's EXPERIMENT or CONTROL variant.
The native apps use the same contract. They retrieve flag information from the Rails backend, and their own tracking hook triggers the same server-side tracking path when native feature code first reads a flag. For server-rendered paths, the flag check itself triggers tracking unless it runs in a skip-tracking context.
Decoupling assignment from tracking adds a network cost. The web and native clients must fire a separate GraphQL mutation back to the server to record the exposure.
The assignment's once-only tracking method is simplified here to show its concurrency control:
1def track_if_needed!2 return false if tracked_at.present?34 with_lock do5 return false if reload.tracked_at.present?67 send_to_mixpanel8 update!(tracked_at: Time.current)9 end1011 true12end
The unlocked first check avoids taking a row lock for assignments already tracked. The lock and second check prevent concurrent requests from sending duplicate events.
Exposure tracking belongs at the shared decision point before control and experiment code diverge. A call inside the experiment component omits control exposures, while a call after experiment-specific navigation selects users based on an effect of the feature. Either placement biases the comparison.
Rollout management
Each feature flag's admin drawer has a Smart Rollout tab. Starting a rollout locks the flag's other gates. This prevents configuration changes from altering populations during the comparison.
During a rollout, a developer can raise the enrolment percentage or complete the rollout. Raising enrolment adds actors to both groups. Current participants keep their original assignments. Success enables the feature for everyone, while Issues disables all gates.
Completed rollout records are immutable. A later test on the same flag creates a new rollout record under a name such as <key>_rollout-2. Administrative actions also write to an audit table, and completed rollout records remain available after flag deletion.
Adoption and observed results
Adoption is voluntary, and teams continue to use custom experiments for multi-variant tests. The chart ends with the last complete quarter: it contains 61 Smart Rollouts through Q2 2026. Another 9 in July brought the total to 70.
Old experiments and Smart Rollouts
We have one documented regression case. It shows the kind of issue a cross-channel comparison can expose.
A team used Smart Rollouts while redesigning Fullscript's patient catalog landing page. The change was presentational, so total order volume was expected to remain flat. Web orders in the experiment group were 4% higher than control.
When the team compared web and native metrics in Mixpanel, total orders across channels remained flat while native app orders fell 10%. The redesign had omitted the deep link that opens the native app when a patient follows a catalog URL on a phone. Orders shifted from the app to the web. Total volume remained flat.
The team completed the rollout as Issues, restored the link, and started <key>_rollout-2. The second rollout showed flat order volume across the web and native app.
Current limits
Smart Rollouts fixes allocation at 50/50. Equal groups fit our binary release-validation use case.
Mixpanel provides the statistical analysis used to validate a release. Smart Rollouts leaves primary metric selection, sample duration, and stopping criteria to each team. Correct assignment and exposure data can't compensate for weak evaluation choices.
When to apply this pattern
This pattern fits a feature-flag system with deterministic bucketing, binary features, stable actor identifiers, and an analytics tool that compares named cohorts. The implementation has four parts:
- A percentage gate selects the enrolled population, and an independently salted hash splits that population into equal groups.
- A persistent assignment records one group per rollout and actor, with a safe off state when evaluation fails.
- Bulk flag evaluation creates assignments without recording exposure, and each client records exposure at the same first-use decision point.
- Configuration stays locked during an active rollout, and completed rollout records remain immutable.
A plain percentage rollout is enough when the goal is limited blast radius without a behavioural comparison. Custom logic or a hosted experimentation platform is a better fit for multi-variant tests, anonymous traffic without stable identifiers, unequal group sizes, automated stopping rules, or a much larger experiment program.
Conclusion
Smart Rollouts adds binary release experiments to Fullscript's Flipper workflow. It combines deterministic group assignment with first-use exposure tracking. The dual-hash split keeps the comparison groups stable as requests race and enrolment grows; the second mechanism keeps Mixpanel cohorts tied to actors who reached the feature. The pattern remains bounded by the quality of each team's evaluation and the fixed 50/50 design.
