Building Fullscript Assist
At the end of June 2026, we rolled Fullscript Assist out to 100% of the platform. This post covers what it is and how we built the system behind it.
What is Fullscript Assist
Before we built Fullscript Assist, our providers had no effective AI option inside Fullscript. They either avoided AI altogether or stitched together their own workflows: Notion databases, custom GPTs, NatMed Pro in one tab, Lexicomb in another, and their EMR in a third. Every provider we talked to had spent years building their own knowledge bank and their own way of practicing: copy-pasting protocols, maintaining supplement stacks, and writing clinical notes by hand.
When we described what we were building, the pushback was immediate: "Why would someone use this over Claude or ChatGPT?"
Because Assist works where providers already practice. It knows the products in your Fullscript Catalog, has access to your patient context, understands your clinical workflow, and is HIPAA compliant. General-purpose AI can’t do that. It doesn’t know your patient panel, can’t add products to a treatment plan, and, in most cases, can’t safely accept protected patient information.
That's what Assist is: A clinical AI assistant built inside Fullscript, to support our providers the way they practice.
What it does
At launch, Fullscript Assist can:
- Research supplements and recommend them with citations from clinical sources
- Check drug-supplement interactions and nutrient depletions via NatMed
- Go from visit notes to a full treatment plan in seconds
- Search Fullscript's knowledge base for platform support questions
- Personalize recommendations to how each provider practices: their preferred brands, their clinical voice, their patient population
Today, Assist is available to more than 120,000 providers on Fullscript. Early feedback confirmed we were solving a real problem. After Assist generated a treatment plan from visit notes, one provider replied:
"Wow! Thank you! This would have taken me close to an hour to create."
Version 1: Why We Rebuilt
Assist Classic launched in early 2026 and reached 100% of providers by April. It was a Python serverless function sitting behind our Rails monolith, using LangGraph for agent orchestration. Providers used it, adoption was meaningful, and it proved there was real demand for AI built into the clinical workflow.
But it also exposed the limits of our architecture.
In LangGraph, adding a new capability meant adding a new code path to a directed graph. The agent wasn't deciding what to do next. It was following a route we'd pre-defined. That approach worked when workflows were predictable, but clinical conversations aren’t. As providers asked more nuanced questions, we found ourselves constantly adding new branches to handle edge cases. The graph approach doesn't generalize.
It also introduced latency problems. Each request was a long SSE stream proxied through the Rails monolith to a stateless serverless Python function—not what our Rails' transactional request model was built for. That contributed to our slow-request problem early on. The Python service didn't run a ReAct loop; it emitted markup we parsed and used to enrich responses via API calls. We didn't have a good story for stateful conversations either: because it was a stateless serverless function, we had to re-hydrate context on every turn.
What we wanted was a true ReAct loop: a model reasoning in a loop, deciding which tools to call and in what order, handling variation to solve our users' problems. We wanted to focus on creating building blocks to get emergent behaviours. That's a different architecture, and it was worth starting over to get right.
Why Elixir, not Rails?
An AI turn isn't a short HTTP request. It's a reasoning loop that might make five or six tool calls over ten to fifteen seconds, streaming partial results the whole time. The natural model for that is a long-lived, supervised process that runs for the life of a conversation. Rails controllers aren't that. You can bolt concurrency onto Ruby, but the BEAM (the Erlang VM, which Elixir runs on) is purpose-built for it: lightweight processes, supervision trees, fault tolerance baked into the runtime. The concurrency model is the right primitive for this problem.
There was a cost: our team had almost no production Elixir experience, and the infra learning curve—not the language itself—was where we spent the most time. Agentic development and a strong team made the language ramp shorter than you'd expect. We'd make the same decision again.
So we built a standalone Elixir/Phoenix application, the Orchestrator, that runs alongside our Rails monolith. The Orchestrator handles the agent lifecycle. Rails stays the source of truth for data.
Jido is the Elixir agent framework we built on. It gave us the ReAct loop, a first-class action model that maps cleanly to tools, a plugin system for cross-cutting concerns, and a skill architecture we'll get to below.
More importantly, the fundamental thing Jido gave us is how agent processes live. Each agent is an Erlang process. It's spawned when a provider starts a conversation, supervised so it restarts on crash, and registered in a process registry so it can be looked up by session ID. That process runs the ReAct loop, holds the full conversation context in memory, and streams results back in real time over Websockets.
When a turn completes, the agent hibernates. Its full conversation state is serialized as a checkpoint to the Orchestrator's PostgreSQL database (the Orchestrator runs its own database; Rails uses MySQL). The process exits. When the next message arrives, the system checks if the process is running, and if not, reads the checkpoint back from the database and starts a fresh process pre-loaded with the full conversation history. The agent picks up exactly where it left off. Providers can close the tab, come back an hour later, and continue mid-thread.
Getting this right took real work. We built a turn persistence layer to accumulate the raw signal stream during a turn and flush structured blocks on completion. We hooked into Jido's plugin system to trigger hibernation at the right moment. Transient LLM failures get automatically retried once inside the process, invisible to the frontend.
The config module holds the tools list, skills list, base prompt, and model configuration. The agent says: use this config, attach these plugins. Jido and the BEAM handle spawning, supervising, hibernating, and thawing.
Authentication
The Orchestrator doesn't maintain its own session system. When a provider connects over WebSocket, the browser's existing session cookie travels with the connection. The Orchestrator proxies that cookie to a Rails endpoint to resolve who the user is. From that point on, every tool call the agent makes goes through Rails with the provider's real session context: the same access controls and authorization logic that govern everything else they do on Fullscript.
The agent only has access to what the provider has access to. The security model isn't something we built custom for AI. It's the same model the whole platform runs on, and the agent inherits it.
This matters practically because it means we can add new tools to the agent without building new auth plumbing for each one. Patient search, product search, lab tests, interaction checks, treatment plan operations: they all go through the same shared authentication primitives. Adding a capability is adding a tool, not a new security layer.
All protected health information, message blocks, conversation history, checkpoints, are encrypted at rest with AES-256-GCM field-level encryption before it touches the database.
The Skill Architecture
As the agent's capabilities grew, we hit a real prompt engineering problem. You can't put everything in the system prompt. A long, dense prompt means the model loses track of instructions at the edges, hallucinates tool behaviors, and gets confused about what to use when. But complex workflows need detailed instructions.
Our answer is the skill system.
Skills are on-demand instruction bundles. The system prompt carries only a compact index: one line per skill, name and short description. The system prompt also instructs the model that when it needs to execute a complex workflow, it should call load_skill with the skill name first. When the model decides it needs to do something like propose a treatment plan or look up a drug interaction, it calls load_skill. The full instructions are injected into the conversation history as a tool result, detailed workflow guidance, exactly when the model needs it.
1SKILLS2- propose-plan: Plan proposal workflow -- builds structured treatment plan proposals...3- interaction-depletion-check: Drug and supplement interaction lookup workflow...4- knowledge-base-answering: Fullscript support and knowledge base answering guidelines...
The model loads the skill, reads the instructions, and executes the workflow. For common entry points like a "Build a Plan" button, we eager-load the skill before the first turn so the model has what it needs immediately with no round-trip.
Skills are also feature-flag aware. A skill can be gated behind a flag. When a session starts, capabilities are filtered to what's enabled for that provider's account. A disabled skill never appears in the index at all. The model is never told it exists. Because skill instruction bodies can be flag-conditioned too, we can roll out behavioral changes to a subset of providers before full release.
This keeps the system prompt compact, makes the agent's behavior auditable (you can read exactly what instructions it loaded for any given turn), and lets us add new capabilities without touching the core agent.
When a provider sends a message, it travels over a Phoenix WebSocket alongside context signals: the current patient, the active treatment plan, and whatever the provider has open on that page. We prepend this context to the prompt as structured XML before it reaches the model. We chose XML rather than JSON because it handles nested context clearly and degrades gracefully in plain text without requiring escaping. The Jido ReAct loop takes over: call the LLM, receive tool calls in the response, execute them, feed results back, repeat until done.
The agent accesses Fullscript data through a MCP (Model Context Protocol) built for internal use. We run an MCP server inside the Rails monolith that exposes platform data as callable tools: patient records, product catalog, lab tests, interaction checks, treatment plan operations. The Orchestrator calls these tools over HTTP on behalf of the agent. The data and business logic stay in Rails. The agent decides when to call them.
The practical value of using MCP as the interface layer is that adding a new tool to the agent means implementing it as an MCP resource in Rails. No new protocol, no new plumbing. The agent discovers it automatically.
On the Rails side, tools auto-generate their JSON schema from the existing .gql query file:
1class ProductSearch < Tools::Graphql::Base2 tool_name "product_search"3 description <<~DESC4 Search for supplements and other lifestyle products in the catalog.5 After getting results, use availableOrMasterVariant.id in recommend_supplements.6 DESC78 generate_schemas_from_query9 annotations(read_only_hint: true, destructive_hint: false, idempotent_hint: true)10end11
The corresponding Elixir proxy is a thin wrapper that speaks JSON-RPC to the Rails server:
1defmodule Orchestrator.Actions.MCP.ProductSearch do2 use Orchestrator.Actions.MCP.Base, tool_name: "product_search"3end
That's the full extent of what you write to add a tool on the Orchestrator side.
Typed signals and rich components
The agent doesn't stream text and leave the UI to parse it. It emits typed signals that drive specific frontend components. When the model has a supplement recommendation to surface, it calls a tool that produces a structured recommendation result. The frontend receives a typed event and renders the appropriate component: a card with product options, dosing, and a one-click add-to-plan action. The same pattern applies to lifestyle recommendations, citation panels, knowledge base results, and thinking summaries.
The frontend maps each incoming event to a typed dispatch:
1const fsAssistEventMapper = (event: AgentEvent): ChatAction | null => {2 switch (event.kind) {3 case "llm_delta":4 return { type: "APPEND_TEXT_BLOCK", payload: { text: event.data.text as string ?? "" } };56 case "recommend_supplements":7 return {8 type: "ADD_SUPPLEMENT_RECOMMENDATIONS_BLOCK",9 payload: {10 supplements: normalizeRecommendationSupplements(event.data.supplements),11 },12 };1314 case "state_changed":15 return {16 type: "SET_PENDING_INTERACTION",17 payload: {18 pendingInteraction: event.data.pending_interaction as PendingInteraction | null ?? null,19 },20 };2122 // 20+ more cases23 }24};
The agent's output is a data stream with a schema. The frontend knows what each event type contains and renders accordingly. This is what makes the interface feel like a product rather than a chat window.
Interaction patterns
Some agent outputs pause the loop and wait for the provider to act. When the agent needs more information, it surfaces a question card: single-choice, multi-choice, or a patient picker when no patient is in context. The provider answers, and the response arrives as a new turn.
On the output side, when the agent has built a treatment plan it renders a full proposal card for the provider to review and accept or dismiss. Accepting triggers the system to create or update the draft plan directly.
Both patterns share the same underlying mechanism. The agent emits a pending interaction signal, the turn ends, and the resolution travels back over the channel when the provider acts. Both the proposal and the resolution are persisted, so the conversation history is a complete record of what was suggested and what happened.
Observability
A ReAct loop is harder to observe than a normal web request. You have tool call sequences, token counts, model selection, latency per step, and failure modes that compound across turns.
We emit structured events at each phase of the loop: turn start, skill load, tool call, tool result, LLM call, turn complete. These go to our standard observability stack. The important thing is capturing the full turn context on error: which model was running, which tools had been called, what the skill state was. When something goes wrong in production, we need to be able to reconstruct what the agent was doing.
We persist the full message history for each turn, so when something goes wrong you can reconstruct exactly what the model saw, the system prompt, loaded skills, tool results, and LLM calls.
Evaluation
Evaluation isn’t optional. We built an evaluation harness with scenario definitions, mocked tool responses, and an HTTP API for running single-turn tests against the live agent. Each scenario defines a provider context, a user message, the expected tool calls, and the expected output characteristics.
Our medical innovation team owns the clinical scenario set. These aren't synthetic engineering test cases. They're based on real clinical workflows and provider behavior. They’re what we run every agent configuration change against. We also have benchmarks we're measuring ourselves against; more on that as the system matures.
The next layer we're building is evaluation against live and synthetic data. That work is still early, but the goal is clear: move beyond pass/fail testing on scripted scenarios toward continuous measurement of clinical quality. The harder challenge isn’t building the evaluation framework but defining what “correct” means when multiple clinicians may reasonably recommend different protocols. Reaching alignment between engineering and the clinical teams often takes longer than the implementation itself.
What We Learned
None of this went smoothly. Here's what actually taught us something.
Failover needs to be automatic. We built a health tracker backed by an in-memory store that monitors per-model error rates in a sliding window. When a model is degraded, traffic is automatically routed to the next available profile. Today, we run Gemini through Cloudflare's AI Gateway with Claude Sonnet on AWS Bedrock as the fallback. Along the way, we uncovered a subtle bug: routing through the gateway caused the model’s reasoning context to be silently dropped on subsequent turns because of a mismatched source identifier in the response builder. That class of bug lives at the intersection of model, SDK, and infrastructure. You don't know to look for it until it bites you.
Kubernetes deployment has different failure modes than a stateless web app. When Kubernetes signals a pod to shut down, in-progress LLM turns need to finish and persist before the process dies. That meant configuring pre-stop hooks, shutdown grace periods, and lifecycle management so the BEAM had enough time to drain. You can't just terminate a process that's mid-sentence with a provider.
Adopting a new ecosystem always comes with a learning curve. New tooling, new operational patterns, and deployment models that behave differently than anything the team had run in production before all take time to absorb. What made the difference was AI coding agents. Having a coding agent that could explain BEAM semantics, suggest idiomatic Elixir, and reason through unfamiliar patterns in real time meaningfully reduced the barrier to entry. We think that's broadly true: AI coding agents have changed what it costs to adopt a new language or ecosystem. Elixir and its ecosystem have been a joy to work with, and we'd make the same call again.
Evaluation is harder than the engineering. We have a harness. The work ahead is tying it more tightly to real production use cases and validating our assumptions about how providers actually use Assist. The longer we're in production, the better our scenario coverage will get, and that will inform how we evolve the harness. One thing that's become clear: when you're building in a domain you're not an expert in, you have to invest in bringing those experts into your evaluation process. Engineers can build the scaffolding. Defining what good looks like requires people who actually practice.
The Principles That Held Up
Looking back, a few bets paid off clearly.
Run agents as long-lived processes, not request handlers. The BEAM process model gives us stateful, supervised agents that hibernate and thaw cheaply. This is the right primitive for conversational AI. A stateless request model would make the architecture significantly more complex.
Keep your data where it lives. MCP lets the agent reach into the existing platform rather than pulling data into a new layer. Adding a new capability is adding a tool, not a new data store.
Inherit your platform's auth model. Proxying the user session through to the agent's tool calls meant we got authorization without building a custom AI security layer.
Don't bloat the system prompt. On-demand skill loading keeps prompts compact and makes agent behavior auditable. Lazy-loading instructions scales as capabilities grow.
Typed signals over text. Structured event types from the agent enable purpose-built UI. The output being a data stream rather than prose is what makes rich, actionable interfaces possible.
What's Next
The vision is simple: everything you can do in Fullscript should be possible through Assist. Every workflow, every domain, every action; available conversationally, in context, without leaving the flow of a clinical encounter.
We’re still a long way from that vision. The next challenges are scaling contributions across a larger team without prompt instructions colliding, maintaining meaningful evaluation coverage as the scenario set grows, and expanding the UX surface to support a much broader range of interactions.
The more ambitious opportunity extends beyond Fullscript itself. Providers don’t solely live in Fullscript. They move between EHRs, practice management software, lab portals, patient communication tools. Bringing data from those systems into a single clinical conversation, and eventually enabling actions across that software stack, is where this becomes truly powerful. MCP as an open standard gives us an architecture to do it. The work is in the integrations.
Questions about the architecture? Reach out.