Scaling Shopify's maintenance_tasks (Part 2): Real-Time Progress for the Gem

Part 2 of 3 on what we built on top of Shopify's maintenance_tasks gem. Part 1 covered the custom UI, and  covers observability and automated cleanup. If your team runs data migrations or one-off scripts in production, there may be something here worth stealing.

Part 1 ended on a promise: the progress bars in our UI update live, and they are not polling. This post is how, and more broadly how we layered real-time and cross-cutting behavior onto a gem we don't own without forking it.

You can add real-time and cross-cutting behavior to a third-party engine you don't own without forking it. Prepend small modules at the right seams, centralize shared behavior in a base class, and isolate heavy work onto its own queue. The skill is choosing seams that survive upgrades, and guarding them so you find out the moment they don't.

The challenge: a great engine with three gaps

maintenance_tasks is great at running the work, but it leaves three gaps we needed to close:

• No live feedback. The gem persists run progress to its table, but nothing pushes that state to a connected browser. Out of the box, the only way to answer "is it still running?" is to refresh and read the latest row.
• Nowhere for cross-cutting concerns. We wanted structured logging, Slack notifications, and metrics on every run. The naive way is to drop the same lines into every task, and across dozens of domains that is how rot starts.
• Heavy work competes with application work. A big backfill is greedy. Run it on the same queue as latency-sensitive jobs and it can starve them, turning a routine data fix into an incident for unrelated features.

The constraints we had to respect

The interesting part of this problem is not the features. It is the constraints we had to honour while adding them.

• We don't own the gem, and we have to support its upgrades. A fork was off the table, so whatever we did had to layer on top of code that could change underneath us.
• Rails reloads classes in development. Any patch we apply has to reattach cleanly every time the code reloads, and it must not stack itself twice.
• Progress ticks fire rapidly. Whatever we hang off the gem's persistence path runs often, so it has to be cheap and it must never take down the actual task if it fails.

The solution

Three changes close the gaps: broadcast progress by prepending the gem, centralize behavior in a base class, and isolate heavy work on its own queue.

Real-time progress by prepending the gem

The gem's Run model persists state through a few methods: persist_progress on each tick, persist_transition on status changes, and persist_error on failure. Those are the exact moments the run's world changes, so they are exactly where we want to tell the browser. We prepend a module onto MaintenanceTasks::Run, which inserts ahead of the class in Ruby's method lookup chain, so our code runs first and super calls back into the gem's original. We let the gem do its real work, then broadcast, without replacing a line of its implementation.

1module MaintenanceTasksRunBroadcast
2  def persist_progress(ticks, duration)
3    super.tap { broadcast_run_update }
4  end
5
6  def persist_transition
7    super
8    broadcast_run_update
9  end
10
11  def persist_error(error)
12    super
13    broadcast_run_update
14  end
15
16  private
17
18  def broadcast_run_update
19    AnyCable.broadcast("maintenance_tasks:#{task_name}", run_snapshot.to_json)
20  rescue => e
21    Rails.logger.warn("broadcast failed: #{e.class}: #{e.message}")
22  end
23end
24
25Rails.application.config.to_prepare do
26  unless MaintenanceTasks::Run < MaintenanceTasksRunBroadcast
27    MaintenanceTasks::Run.prepend(MaintenanceTasksRunBroadcast)
28  end
29end
Copy

Two small things matter more than they look. The broadcast is wrapped in a rescue, because a notification must never fail the task. And the patch is applied inside to_prepare, which runs on boot and after each reload, with an unless ... < Module guard, so a reload cannot stack a second copy of the module and turn one broadcast into many.

On the browser side, a React hook subscribes to one channel per task and updates state as messages arrive. It also fails quietly. If the socket cannot connect, it shows a small "live updates unavailable" notice and the UI keeps working through ordinary reads, because a missing WebSocket should never block running a task.

1consumer.subscriptions.create(
2  { channel: "MaintenanceTaskRunChannel", task_name },
3  { received: (data) => {
4      const msg = typeof data === "string" ? JSON.parse(data) : data;
5      isLogUpdate(msg) ? onLogUpdate(msg) : onRunUpdate(msg);
6  }}
7);
Copy

We broadcast through AnyCable, a Go-backed server that speaks the Action Cable protocol but handles far more concurrent connections than the default Ruby implementation. Because it is protocol-compatible, the channel and the client subscription look just like Action Cable. End to end, a single tick travels a short path: the worker calls persist_progress, our prepended module broadcasts a snapshot through AnyCable, and the subscribed React hook updates the progress bar.

One base class for everything cross-cutting

Every task inherits from a single ApplicationTask instead of the gem's MaintenanceTasks::Task. The base class registers lifecycle callbacks once, and they fan out to the concerns Part 3 covers: durable logs, Slack notifications, and metrics.

1class ApplicationTask < MaintenanceTasks::Task
2  after_start     { notify(:start) }
3  after_complete  { notify(:complete) }
4  after_error     { notify(:error) }
5  after_pause     { notify(:pause) }
6  after_interrupt { notify(:interrupt) }
7  after_cancel    { notify(:cancel) }
8end
Copy

The value is not any one callback. It is that "what happens when a run starts, finishes, or fails" is answered in exactly one place. The base class paid for itself the moment we added the second cross-cutting concern, because that one was a few lines in a file that already existed instead of a change rippled across dozens of tasks.

A base class only helps if every task uses it, and "remember to inherit from the right class" is not a strategy. The RuboCop cop from Part 1 enforces it, which is what lets us reason about lifecycle behavior across the whole system at once.

Keeping heavy work in its own lane

Maintenance work runs on its own dedicated Active Job queue, on Solid Queue, so a big backfill cannot starve latency-sensitive jobs. We point the gem at a one-line job class that just sets the queue name. Alongside it sits some easy-to-forget schema work: a metadata column on the runs table for who triggered a run (Part 3), bigint tick counters because real backfills overflow a plain integer, and an index on task name with status and creation time so listing recent runs stays fast.

Tradeoffs

Prepend was the fast, surgical middle path between a fork (total control, permanent maintenance, upstream drift) and an upstream pull request (cleanest when broadly useful, but slow and not guaranteed to land). The cost is real. We couple to method names the gem treats as internal, and an upgrade could rename persist_progress and silently break the broadcast. We mitigate with a pinned gem version and a test that asserts the seam still fires, so a moved seam fails the build instead of a user noticing dead progress bars. That test is the whole reason patching private internals is acceptable here. The other lesson worth repeating is that reapplication and idempotency are not optional, since skipping to_prepare or the guard gives you broken reloads or duplicated broadcasts.

What it got us, and when it's worth it

The observable change is straightforward. Progress is live, so the honest answer to "is it working?" is visible on the screen instead of guessed at through refreshes. Cross-cutting behavior is defined exactly once, so adding the next concern is a small edit in a known place rather than a migration across the whole task suite. And big backfills no longer pull latency out from under unrelated jobs.

The pattern travels if you are adding live progress to your own background-job system: broadcast only at the moments state actually changes, use one channel per task with a client that degrades gracefully, centralize cross-cutting behavior in a base class and enforce it with a linter, and isolate heavy work on its own queue. The one part specific to us is the prepend, because the gem gave us no official hook for broadcasting. If the tool you are extending already broadcasts or exposes a real hook, skip the prepend and the upgrade risk that comes with it.

Conclusion

Three changes turned a gem that broadcasts nothing into a live system: prepended broadcast hooks, one base class, and an isolated queue. Running a maintenance task no longer means kicking it off and refreshing to guess at progress. You watch it happen.

The lifecycle callbacks we wired into the base class do not just fire and vanish.  follows them into structured logging, Slack alerts, metrics, and an automated system that files tickets to delete dead tasks.