Building Fullscript Assist, Part 2: Keeping the Context Small

How we treat the context window as a scarce, degrading resource, and the three techniques we use to spend it carefully.

In Part 1, we covered what Fullscript Assist is and the architecture behind it: long-lived Elixir agent processes on the BEAM, Jido for the ReAct loop, MCP tools into the Rails monolith, and an inherited platform auth model. If you haven't read it yet, start there.

This post goes deep on one problem that shapes almost every design decision in the Orchestrator: the context window.

The Context Window Is a Budget, Not a Bucket

It is tempting to think of the context window as a bucket. You have a number of tokens, you pour things in, and you only worry when it overflows (tokens hit the context limit). That thinking is wrong in two ways.

First, models degrade well before the hard limit. Attention gets thinner as the window fills. Instructions at the edges get lost. The agent starts to hallucinate tool behaviors, forgets what it decided three turns ago, and gets confused about which tool applies when. We saw this directly as Assist's capabilities grew: a longer, denser prompt made the agent less reliable, not more. The useful window is smaller than the advertised one.

Second, every token you spend is a token you cannot spend later. A clinical conversation might run for twenty turns across an hour. If the first research call dumps 40KB of raw retrieval into history, that impacts every subsequent turn, and you pay for it in latency and, more importantly, in quality.

So we treat context as a budget. The guiding question for every feature is not "does this fit?" but "is this worth the space it takes on every future turn?" Three techniques fall out of that question, arranged from cheapest to most expensive:

1. Do not put it there in the first place. On-demand skills, capability gating, compact structured context.
2. Isolate the expensive work. Sub-agents that keep their own churn out of the main thread.
3. Reclaim space when you are running low. Compaction: summarize old history to buy back the budget.

Let's walk through each one.

1. Do Not Put It There in the First Place

On-Demand Skills

We covered the skill system in detail in Part 1, so we will not rehash the mechanics here. The short version, seen through the budget lens: the system prompt is the most expensive real estate in the whole system, because it’s included in every single turn. So it carries only a compact index of skills, one line each, and the full instruction bodies load on demand as tool results exactly when the model needs them. A skill body that might be a thousand tokens costs one line of index until the moment it is actually used.

A skill that is turned off costs zero tokens. Skills are feature-flag aware, so a disabled skill never appears in the index and cannot be loaded. The same gating applies to tools: disabled MCP tools are unregistered from the live agent, so their JSON schemas never consume prompt tokens for that session either. A leaner capability set is a leaner prompt.

Compact, Structured Context Instead of Raw Dumps

When a provider sends a message, it arrives with context signals: the current patient, the active treatment plan, whatever they have open on the page. We could serialize all of that verbatim, but instead, we prepend it as compact structured XML, and we cap the parts that can run long. 

In FS Assist, one example that clearly stands out is the active treatment plan: instead of serializing every product and lab test recommendation along with its attributes, we inject a short snapshot that is truncated to a few hundred characters. The agent gets "here is the current plan outline" without carrying a full object on every turn. The agent has tools to fetch the missing data when a turn actually needs it, so depth is paid for on demand instead of rented for the rest of the conversation.

Multimodal uploads follow the same instinct. We do not store file binaries in the conversation checkpoint. We store a short reference marker, a handful of bytes, and resolve it to the real bytes from S3 only in the moment before the LLM call that needs it. The history stays small, both in the context window and on disk.

2. Isolate the Expensive Work: Sub-Agents

Some tasks are inherently messy. Research was the clearest example for us. Answering "what does the evidence say about magnesium glycinate for sleep?" means refining a query, searching a curated evidence index, reading back a pile of candidate sources, reasoning about which ones are relevant, and discarding most of them. That is many tool calls and a lot of intermediate reasoning, and almost none of it is worth keeping.

If the main agent did that work inline, its history would be filled with search noise: eight retrievals, dozens of candidate chunks, the model thinking out loud about relevance. All of it would then be included on every subsequent turn of the conversation, for a single citation panel.

So we spawn a specialized sub-agent. A dedicated research agent runs its own ReAct loop, in its own short-lived BEAM process, with its own budget. The main agent calls one tool, spawn_research_agent, and gets back only a compact structured result. That result is what lands in the parent's tool history. The child's churn never touches the parent's context.

What comes back looks like this:

1{:ok,
2 %{
3   research_bundle_id: Ecto.UUID.generate(),
4   sources: sources,
5   planner_reflection: planner_reflection
6 }}
Copy

Three fields, and nothing else. research_bundle_id identifies the bundle for persistence and the UI. sources is the curated evidence the parent should cite, capped at eight. planner_reflection is a short note from the research agent about how it searched, not a literature summary: something like "Ran 3 searches covering efficacy, dosing, and pregnancy safety." The parent needs that for progress UI and for deciding what to do next.

That is the whole trade. The parent's history gains one clean tool result. The forty candidate chunks the child waded through, the query refinements, the relevance reasoning: gone, because they lived and died in a process the parent never saw.

The way we assemble that payload matters. Models are bad at faithfully echoing large structured objects, and asking them to do it would burn the child's budget for no gain. So retrieval results bypass the LLM: the search tool writes them into a short-lived collector process whose pid is threaded through the tool context. The child returns only selected source ids and planner_reflection. On completion, the spawn action drains the collector, matches ids to sources, and builds the map. The model judges; the plumbing moves the data.

The plan-builder sub-agent works the same way. Turning visit notes into a structured treatment plan is a multi-step workflow with its own instructions and its own tool calls. It runs as an isolated child and hands the parent back a validated intervention envelope, or a "needs clarification" object if it could not proceed. The parent thread stays a clean record of the conversation, not a transcript of how the plan got built.

The general principle: when a task involves a lot of throwaway reasoning, give it its own context and let it die there. The parent only pays for the answer, not the work.

3. Reclaim Space When You Are Running Low: Compaction

The first two techniques keep new content small. But a long conversation grows no matter how disciplined each turn is. Twenty turns of legitimately useful exchange will eventually crowd the window. When that happens, we buy the budget back.

Before every turn, a synchronous check runs. We sum the token counts of the conversation entries since the last compaction anchor, and compare against a threshold. Today that threshold is about 80% of the model's advertised context limit. We trigger early on purpose: the useful window is the quality budget, not the max on the spec sheet.. Below it, the check returns :skipped cheaply and the turn proceeds untouched. That is the common case, and it is close to free.

When we cross the threshold, we compact. The important part of that design is not the summary. It is what we refuse to summarize.

We keep the most recent turns verbatim: today, the last two full turns, including the user's messages, the assistant's replies, and the tool traffic (calls and results) from those turns. Everything older gets folded into a summary. From the model's point of view, the conversation becomes a compact [Summary of earlier conversation: ...] followed by those recent turns in full.

That verbatim tail is load-bearing. A summary is good enough for older context: which patient was discussed, what was recommended, what questions are still open. It is a poor substitute for the turn you are about to continue. Providers often follow up on what just happened: "add that to the plan," "why that dose?," "check interactions for the last two." Those references only resolve cleanly if the model still has the exact recent exchange, not a paraphrase of it. Summarizing the tail would save a little space and lose the coherence that makes the next turn work.

We learned that the hard way. Our first compaction design summarized everything older than the current turn, including the exchange the provider was about to continue. In long sessions that worked fine for "what did we discuss last week?" It failed on those immediate follow-ups. The summary still knew magnesium had been discussed. It did not retain which recommendation, at what dose, from which tool result. Assist would re-ask or invent a slightly different version of what it had just said. Keeping the last two turns verbatim fixed it. The summary remembers what happened. The tail remembers what "that" means.

So the split is deliberate. Older history is compressed into facts. Recent history stays literal. Subsequent compactions fold the previous summary forward and advance the verbatim window, so the summary itself never becomes the thing that bloats, and the model always has a sharp view of the last couple of turns.

The summarization itself runs on a fast, cheap model with a focused prompt:

1You are summarizing a conversation between a user and an AI healthcare
2assistant to preserve context while reducing token usage. Write a concise
3summary of the conversation below that captures the key facts, decisions,
4and context needed to continue the conversation. Focus on: patient details
5mentioned, health concerns discussed, recommendations made, and any open
6questions. Write in third person. Be factual and specific.
Copy

Mechanically, we write a new context snapshot: the summary plus the verbatim tail. That snapshot is what the model sees on the next turn, and what we reload when a hibernated agent thaws. In Jido terms, we append an ai_context_operation with type :replace. Jido's projection layer applies it automatically, so compaction does not need a separate read path.

Two properties make this safe to run in the hot path. It is synchronous but lazy: the expensive summarization only happens on the rare turn that crosses the threshold, and the check is trivial otherwise. And it is fail-open: if the summarization call fails, we log it and let the turn proceed with the un-compacted context. A degraded summary model can never block a provider from getting an answer. The worst case is that we run closer to the limit for one more turn.

One History

We considered treating conversation context like a cache you can evict, and like a knowledge base you can query. Both weaken the thing providers actually rely on: a continuous thread.

So the rule is simple. There is one history. Skills and sub-agents keep junk out of it. Compaction rewrites older turns into a summary inside it. Nothing important disappears without a record of what replaced it. When something goes wrong in production, we can reconstruct exactly what the model saw on any turn.

What's Next

The techniques above are how we spend the window today: keep most things out, isolate the expensive work, and reclaim space only when we must, with a summary we can audit.

Context windows will keep growing. That does not make the discipline obsolete. A bigger budget spent carelessly still degrades, still costs latency, and still buries the signal.

What changes next is the blast radius of a bad spend. As more of the team ships skills, tools, and sub-agents into Assist, every addition competes for the same attention budget. The work ahead is less about inventing a fourth technique and more about keeping the budget honest under that growth, including evaluating whether compacted threads still resolve the follow-ups providers actually type.

Stay tuned for Part 3 of the series, where we'll cover how we keep conversations consistent across sessions as we scale Assist.

In the meantime, if you have any questions about FS Assist, reach out!