Using an LLM to find out what our merge requests are actually doing

At Fullscript, we open a lot of merge requests (MRs). Our developers put a premium on speed and are proud of the output that they can produce, especially over the past year as individual contributors have really started to integrate AI in daily development workflows. With over 120 developers contributing daily to our various codebases, the number of MRs being merged daily has never been greater. However, does that mean that AI has helped us deliver more features to our users? Or are we spending all of this time reverting our changes and fixing bugs?

Until recently, we didn’t have a reliable way to determine if the increased output from developers meant that we were providing more value to our customers, or if AI was simply moving us in circles by producing bugs and tech debt, and then helping us patch those holes with possibly more bugs. We needed a way to start tracking what our MRs were doing at a high level.

Why not have developers tag their MRs?

We could have told developers to tag their MRs with a specific category, but even with published guidelines on how to categorize the change, different developers will categorize slightly differently. If an MR changes some behavior for the user, is that a feature? What if the previous behavior was not what the Product Manager wanted to happen, does that make it a bug fix? And let’s not forget that when moving quickly, honest mistakes can happen and people will forget to apply the tag. Then, we’d be left with an incomplete data set, which in some cases could be worse than no data to begin with.

Five categories and a security flag

Instead of putting the burden on human developers, we designed a system to use an LLM to categorize our MRs for us. We settled on a system that would categorize MRs into one of five categories: Feature, Bug Fix, Chore, Maintenance, and Revert. We decided to have separate categories for Chore and Maintenance. We defined Maintenance as changes that keep things current without adding capability, such as dependency bumps, documentation, or security hardening. Chores are structural work that doesn’t add capability and does not fix behavior, such as refactors, renames, or removing dead code.

In addition to its category, each MR gets a security flag to indicate if the change is security-related. If Security had been a sixth category, it would force us to decide if a bug fix related to user permissions gets categorized as Bug Fix or Security. Keeping Security as a separate flag gives us more granular control in monitoring and reporting.

After we internally agreed on the distinction between each of the categories, we created a prompt that included a priority-ordered decision tree that the LLM would use to categorize an MR based on the title, description, files changed, and diff as evidence. The model checks each category in order and picks the first one that fits. We use Chore as the fallback when none of the other categories fits the change:

1. Revert
2. Bug Fix
3. Feature
4. Maintenance
5. Chore

We found that giving the LLM a decision tree to follow produced more consistent results than when we simply described the available categories. We treat MR content as untrusted input so that the model doesn’t follow instructions that get embedded in the title, description, or diff. We also tell the model that the MR author’s own categorization is not a directive. If the author claims that a change is a Chore, but the primary intent fits our definition of Maintenance, we want the model to categorize the change as Maintenance. We’re also well aware of the nondeterminism of LLMs. We took a few steps to keep results consistent over time:

• We started with hand-categorizing 81 MRs from the different categories and iterated on the prompt until we had over 91% accuracy when categorizing the MR and over 98% accuracy on identifying if it was a security-related change.
• Both the underlying model used to categorize the MRs as well as the prompt are versioned in source control config. We currently use Haiku 4.5 and found it results in both high accuracy and speed of categorization. When the time comes to upgrade the model or the prompt, it will involve an intentional change to our config.
• Any changes to the model, prompt, or data from the MR that is given to the LLM for the purpose of categorization need to run through our eval system. If the change results in a lower accuracy than our baseline, that serves as an indicator that we need to reevaluate the change being introduced to our categorization system.

Daily categorization

Once we’d built the categorization logic, we deployed it as a Kubernetes CronJob that runs daily. For the repositories that we’ve specified in config, the job fetches MRs that have been merged within the past 3 days and categorizes any MR that has not yet been categorized by the current prompt version. By over-fetching slightly, we allow the system to self-heal if there was an intermittent failure encountered when categorizing any individual MR. The categorization gets persisted to a database table, along with an identifier connecting the categorization to the MR’s author. This allows us to observe the trends of different teams or the department as a whole.

With this data being surfaced for everyone to view, we have begun to build dashboards and visualize answers to questions such as:

• What percent of our merged MRs are for features?
• Do we tend to ship bug fixes on a particular day of the week?
• Has our revert rate started increasing?
• Does an increase in bug fixes correlate with the release of a particular feature?

One piece of the puzzle

The data we collect from categorization has some limits when viewed without context. For example, does an increase in bug fixes mean that AI is introducing more bugs? Or does it mean that AI has allowed us to move quicker and has given us the ability to act on low-priority bugs that so often get pushed into the backlog to sit forever? The category can’t tell us which one of these is true.

But it becomes more useful when we combine the category with other data. If we have a rise in bug fixes that follows a big feature release, chances are high the feature introduced some regressions. However, a rise in bug fixes while incidents or support requests stay steady may point towards backlog cleanup. Now that we have this categorization, we can combine it with other productivity metrics and start to gauge how successful AI has been in helping us deliver more value to our users.