Human First, AI Empowered - How’s it going?

At a recent team onsite, Staff Engineer Mark Johnson casually mentioned plans to upgrade a project next year to better integrate generative AI. The project's original completion date? January 2025. This timeline perfectly captures how much has changed: in just the last nine months, including GenAI in a project has gone from a "nice-to-have" to "table stakes." This sea change has impacted every corner of our business. I'm incredibly proud of how our team has navigated this shift. Now, let's dive into our journey thus far: What went well, where did we stumble, and what's next?

The first thing we did well was make it someone’s job.  We were lucky to bring in the brilliant Sahar Rahmani to help lead our AI initiatives and build our new GSD (Get... Stuff Done) team. This team is responsible for both enabling the business to self-serve integrating AI into their work and building more complex software that accelerates workflows across teams. She and the team spent the first few months building the foundation for the rest of our programs.    

Their AI work can be lumped into two broad buckets - the culture shift and the actual technology.  The technology will move forward at a breakneck pace whether we like it or not. Learning, embracing and adapting is the order of the day here.  By contrast culture moves much more slowly on its own, but we can have a much greater impact on shaping it.  That’s why, when we built the team, we looked for more than technical skill. We selected engineers who were also personable, collaborative, and user oriented, people who could shape both our technology and our culture. Luckily for us we have a plethora of talent just like that.

If you will permit me an aside that might come across as a little “marketing-y”, the amazing group of engineers we have at Fullscript as a whole is one of our super-powers.  These folks not only create amazing software and systems, but care so damn much.  Our largest points of friction in our culture surveys are often how we can do more, better.  And our engineers come with concrete ideas for improvements, not just complaints.  My job as a leader is more one of wrangling and ordering all the great ideas rather than trying to push an unwilling group forward.  There is no position I would rather find myself in. 

The first task the GSD team took on was figuring out how to help people use these tools confidently in their day-to-day work.  Broadly people fell into two camps - the impassioned and the sceptical.  To be clear both were totally rational places to be 9 months ago.  For the impassioned we created safe places to discuss and share what they were working on, with the aim to inspire people to try new things based on our shared experience.  For the more sceptical we ran extensive training programs for both the technologies we already had available and for the new platforms we rolled out.  We highlighted improvements to those tools as they evolved and established dedicated office hours for people to come and ask questions and learn from others.  We also talked, a lot, about AI in every venue we could think of.  From our townhall meetings, to our weekly North Star Emails, to Slack channels and everything in between.  Even I was sick of the word AI by the end of the blitz.   

These efforts have paid off.  Our ChatGPT usage has gone from 134 active daily users to 560.  Gemini usage has exploded similarly. Tools like NotebookLLM have become invaluable tools.  We have business users building GPTs and workflows in ways I would have never imagined when we set out on this path. More impressively the conversation has genuinely shifted from healthy scepticism to embrace.  Questions have gone from “can AI do this” to “here is how I’m going to improve this process, but I’m stuck on this one thing”.  It’s been truly remarkable.  We still have lots more training and education to do, of course, but Fullscript has really risen to the challenge we asked of them at the start of the year.

Next we set about meeting with people all over business.  Asking them about processes that could be improved. Finding manual work that, while important, could be automated.  Many hours were spent shadowing people while they worked.  Out of these sessions came a huge list of opportunities, more than our tiny team could tackle.  We whittled that list down to a few tangible projects we would work on.  Experiments to see if what we thought would match with reality. While many of these are still a work in progress the first few are shipping and showing the gains we expected, or even better.  Perhaps this is best exemplified by the awesome work we delivered in partnership with our customer support team.  Our first big swing was automating their wrap up notes at the end of each call, and it’s been an amazing success.  It’s driving down post-call time significantly, while also raising accuracy and detail.  So in the rare event someone has to call us back, the next person they talk to has the right context to help them out quickly.  Many more projects have shipped, or will be shipping soon, and I look forward to sharing more details on those efforts next year. 

The GSD team tackled all of this while also building the infrastructure to power this software.  From deploying our workflow tool (N8N), establishing permissions and governance of the various LLM providers, and working with our data team to ensure we have good sources of sanitized data, this was an enormous task. For a tiny team, they  got a lot done, and continue to move with a pace that is impressive. 

But what could we have done better? The two key areas we underinvested in from my view were AI Evaluation and LLM routing.  Much of the development time was spent testing our applications to ensure correctness.  Having a solid evaluation framework would have sped up that work  considerably. Delaying its implementation  was a clear miss on my part.  AI routers, and the visibility into usage and guardrails they provide was also not something I thought was necessary.  I genuinely believed that one model would emerge as the clear market leader over the course of the year, and would drive the bulk of our usage.  Instead the providers continue to leap-frog each other, and there remains a large quality variance model-to-model depending on the task.  For the foreseeable future we will pick the best model for purpose, and adding a router to our stack will make that much easier to manage.  

Our other miss was around messaging at the start.  The effects AI will have on the workplace and society as a whole are still in question.  We rolled out an AI strategy that was overly aggressive, and did not acknowledge that very human feeling that change can be frightening.  It wasn’t until we gained clarity on what AI truly meant for Fullscript and communicated that clearly and authentically that we began to see the pushback ease.  We as leaders owe it to everyone to be open and honest about why we are pushing in a certain direction, and why.  

Despite some missteps along the way, this has been a truly transformational year at Fullscript.  We are genuinely getting more done, more quickly.  But even more importantly the AI revolution has started to shift how we think.  We have shifted away from the status quo.  We are now all thinking about what processes can be streamlined, integrated, improved - AI or not. We are talking more about how to leverage our data, how to better use our systems, and new ways to build.  Yes, we use AI a lot, and that's great. But this change in thinking is honestly the more impressive achievement.