Part 3 of 3 on what we built on top of Shopify's maintenance_tasks gem. Part 1 covered the custom UI, and Part 2 covered the real-time architecture.


Part 2 gave every run an ApplicationTask base class that fires lifecycle callbacks. This post is what we hang off those callbacks. Running the work is the valuable part, and the gem already does it well. What we kept adding is everything that makes those runs trustworthy once a system like this is in daily use.

Running the work is the point, and the gem already does that part well. What turns a pile of one-off scripts into a platform a team can trust is everything around the run: durable logs, alerting, accountability, and a system that fights entropy by nudging teams to delete their own dead tasks.

The problem: everything after the run

Once tasks are easy to run, you run a lot of them, and a new set of gaps shows up:

  • Logs were ephemeral. The gem's output went to standard out, so it vanished when a pod cycled, leaving no durable record of what a backfill changed.
  • Failures were silent. Nothing announced a failed run, so they were found by accident, usually when someone went looking.
  • There were no metrics. We could not answer basic questions about run volume, duration, or failure rate.
  • Runs were anonymous. Nothing recorded who triggered a given run.
  • One-off tasks rotted in the repo. Most are written for a single deploy, run once, and then forgotten, and deleting another team's dead code is nobody's job.

Two constraints shaped the fixes. Observability is best-effort and must never fail a run, and cleanup must route to the owning team automatically and never delete code on its own.

The solution

The observability layer hangs four best-effort sinks off the Part 2 callbacks, and a separate governance job audits the tasks themselves.

The observability layer: one callback, four sinks

When a run starts, completes, errors, pauses, interrupts, or cancels, the base class fans that single event out to four best-effort sinks, so a failure in any of them never takes down the run.

Durable logs. A logger writes each line to a dedicated maintenance_task_logs table keyed to the run, while still tagging the Rails log. Writes are batched for throughput and flushed at lifecycle boundaries so you never lose the tail of a run, and each line is broadcast over the same per-task channel from Part 2, which is how the UI shows logs live.

1def persist_log(level, message)
2  entry = MaintenanceTaskLog.new(run_id:, level:, message:, logged_at: Time.current)
3  @pending << entry
4  broadcast(entry)
5  flush if @pending.size >= BATCH_SIZE
6end

A separate cleanup service deletes logs older than 120 days and sweeps orphaned logs whose task no longer exists, so the table does not become the new problem.

Threaded Slack. The first event for a run posts a parent message. Every event after threads beneath it, and the parent is edited to show the latest status. One run is one tidy thread, carrying who triggered it, the environment, and a link to the task, instead of five loose messages scattered down a busy channel.

Metrics. The same callbacks emit Prometheus metrics through Yabeda: run counts by status, errors by class, duration, items processed, and enqueue-to-pickup lag (created to started), which is usually the first sign the dedicated queue is under-provisioned.

Attribution. The gem's metadata hook records who triggered each run. It is the accountability counterpart to Part 1's secret-masking. Part 1 kept sensitive inputs out of run history, and attribution makes sure every run still has a name attached.

Self-cleaning governance

This idea was not invented here. We borrowed it from another corner of our codebase that already nudges teams to clean up a different kind of dead code, and pointed it at the maintenance tasks themselves. The result audits the tasks and nudges owners to delete the dead ones. It is governance as code with a human always in the loop, because it files tickets and never deletes anything itself.

Most maintenance tasks are one-off migrations: written for one deploy, run once, then they sit there. The pile only grows, and a codebase full of code that looks live but is not is its own hazard.

A weekly cron walks a short pipeline. First it finds stale tasks: load every task class, drop the base classes and the opted-out ones, and keep those whose most recent run is older than 60 days. Tasks that have never run are left alone, because "never run" usually means new, not dead.

1concrete_tasks
2  .reject { |t| t.keep_permanently? }
3  .select { |t| (last = last_run_at[t.name]) && last < 60.days.ago }

Then it resolves the owning team with CodeOwnership, an open-source library that maps classes to teams through metadata already declared in the repo, so there is no separate mapping to maintain. Finally it files a ticket in the owner's tracker, or, if one already exists and has sat untouched for a week, bumps its priority instead of duplicating it.

The opt-out is one line, deliberately non-inherited so each long-lived task states its own intent:

1class RecurringImportTask < ApplicationTask
2  keep_permanently!
3end

Two properties make it work. It is self-healing: closing a ticket does not stop the nudge, because the next weekly run re-files it until the task is actually deleted or opted out. And it escalates: an ignored ticket climbs in priority rather than fading into the backlog.

Tradeoffs

The central choice was tickets, not auto-delete. We could have opened deletion merge requests automatically, but removing code is too aggressive, and deleting something load-bearing by mistake costs far more than a task lingering an extra week. Trust matters more than speed when you touch other teams' code. We also kept logs in our own table rather than only an external aggregator, so they sit beside run data in the same UI, at the cost of owning retention. And every sink rescues and continues, so a dropped alert is possible but telemetry can never break the work.

What it got us, and when it's worth it

The observable change is a different operating posture. Failures now surface in a Slack thread within seconds, attributed to the person who triggered the run, instead of being found by accident. Every run leaves a durable, searchable log and a named operator. And dead tasks get a ticket routed to their owners instead of accumulating silently.

The pattern travels without our stack. Emit lifecycle events from one place, fan them out to durable logs, alerts, and metrics that are all best-effort, and capture who triggered each run. Then, if dead artifacts pile up the way ours did, build a periodic audit that maps them to owners through whatever ownership metadata you already have and files tickets with an explicit opt-out and a human in the loop. Make the nudge self-healing so it cannot be silenced by being ignored.

A clear note on when not to bother. The governance piece only earns its keep with many one-off tasks and clear ownership to route against. With a handful of tasks, a calendar reminder does the same job. The observability layer, on the other hand, pays for itself the first failure it catches automatically.

Conclusion

Across the series we kept the gem's execution engine and rebuilt around it: a UI any engineer can use, a real-time pipeline that makes runs observable, and an operations layer that makes them durable, accountable, and self-governing. Take a solid engine you do not own, and build the product around it that a team can depend on. Persist your logs, thread your alerts, or build a small owner-routed audit for your own dead code. Once a system fights entropy on its own, it keeps working without you.