AI is becoming the first tool we reach for. This is a story of when that tool was right, wrong, and half-right.

The Challenge

At Fullscript our product catalog has tens of thousands of products, each requiring a standardized and regulatory-compliant representation on our website.

All products should have at least three images in the carousel: front, back and the Supplement Facts table. The issue is, oftentimes, only front and back images are made available to the catalog team.

Historically, the solution to this problem was for the catalog team to download each back image, manually crop the Supplement Facts, create a new image asset, set all of the metadata and finally upload the cropped label. This is obviously a very time-consuming and manual process that is prime for automation.

Our Solution(s)

Use a general-purpose model

The first thing we tried was to see how an LLM would handle this type of challenge. Image capabilities have come a long way in the past year, so it seemed as though it had some potential. The results left something to be desired…

The LLM does a good job of identifying where the Supplement Facts table is located, but the trouble is that the LLM is not able to perform a native crop; it generates a brand new image from what it detects is the Supplement Facts section. This results in several discrepancies, such as the format of the label, missing info and hallucinations of some of the text characters.

This option was not going to be viable, as accuracy of the Supplement Facts is paramount.

Train our own model

Plenty of auto-cropping tools can be found, but they are either too generic or have been trained on niche criteria, different from ours. Our use case was quite specific, so there was no out-of-the-box tool that we could leverage. But these tools did prompt us to attempt training our own model to crop images for our use case.

Leveraging Google Vertex, Google’s machine learning platform, we manually annotated a subset of the product images, as a training set for the model. This is a very time-consuming and manual process, although the initial results seemed promising!

But as soon as a curve ball was thrown at the model, the results changed…

These models become accurate by being trained on common patterns in the images and being able to identify those same patterns in new images. The problem is that, although Supplement Facts tables are fairly standardized, there can be variances and the surrounding layout is far from standard. New suppliers and packaging are constantly being added to the catalog. Even if we spent countless hours training our custom model, there would always be the risk of novel packaging it hadn’t seen before.

This option was out of consideration.

Custom Python Solution

The next attempt was a deterministic code solution: identify the Supplement Facts table directly from its visual structure, rather than have a model infer it. Supplement Facts have a predictable pattern: a rectangular box, solid border, high contrast, so we used OpenCV’s edge and contour detection to locate this shape and contrast pattern. Then we used the Pillow library to perform the cropping itself.

This solution seemed promising at first: it did a great job, clean borders, perfect padding and no wasted space. There was only one problem: on some packaging, there are other visual elements sharing this same pattern.

Contour detection can’t tell the difference between a Supplement Facts box and a suggested use box. The natural solution would be to add OCR detection to confirm we had the right box. But there is also variance in the wording of these tables by region and product type, so even this was not reliable.

This solution was getting closer, but was too rigid to work completely on its own.

Final Solution: A Hybrid Approach

With most of our options exhausted, we reviewed the results and realized that two of the solutions actually performed very well, but at opposite ends of the process.

  • At one end, the LLM was exceptional at identifying the Supplement Facts table itself, even when exposed to a diverse set of products.
  • At the other, the deterministic cropping solution performed flawlessly in the crop itself, preserving original quality and content.

This gave us the idea to combine the two solutions. We would use the LLM to identify the Supplement Facts, but instead of returning an image, we would have it return the coordinates. These coordinates would then be used to perform the native crop.

The results were excellent!

Getting the crop right was only the final step in a larger pipeline. We needed to make sure only suitable inputs made it that far in the first place. Round bottles were the first type of image we filtered out; these would not crop easily and required the supplier to submit a new flat version of the label. Low-quality images were the next type of image to be filtered out; these are unlikely to be cropped cleanly and we would want to source a higher quality image from the supplier anyways.

Once an image passes those checks, the LLM identifies the label and returns coordinates, and Pillow performs the crop. But we don’t want to just blindly trust the results that are returned. We use a separate, independent model to score the crop on a scale of 0-1. Anything that does not meet a 90% confidence threshold gets retried, up to two more times, with increasingly powerful models each time. Only once a crop clears this bar, does it move on to a final human review and upload into our catalog.

88 hours saved + ongoing savings

This automation went on to save our catalog team more than 88 hours of manual cropping work to backfill thousands of images. This number will only continue to grow in the coming weeks, months and years with each new product that is added to the catalog.

More importantly, a manual crop job is no longer a bottleneck for a product page being complete. This also has the added benefit of freeing up the catalog team to work on more strategic initiatives.

Other Applications

This approach could be extended further: once the nutritional facts table has been located by the LLM, OCR could be used to extract the values, rather than performing this task manually. It could also be applied to other types of documents where there is a predictable field type but unpredictable layout, such as invoices and contracts.

Conclusion & CTA

We didn’t know when we started this project which solution would be the correct one, but it turns out, no single solution was up to the task. An LLM hallucinated text when asked to act alone, a custom-trained model couldn’t be relied on when presented with packaging it wasn’t trained on, and a deterministic script couldn’t tell the difference between a Supplement Facts table and a Suggested Use box.

The lesson we took from this was not a vote of confidence for or against AI. It was to be more thoughtful about where and when to leverage AI in our solutions. Locating a Supplement Facts table on a diverse set of packaging requires judgement, cropping exact pixels doesn’t. Once you split a problem into these buckets, it becomes clear how to split your solution.