In Part 2, we promised to explain how we keep conversations consistent across sessions as we scale Assist. This post pays that off.
Assist runs each conversation as a live BEAM process. State checkpoints to Postgres, and a hibernated agent thaws from that checkpoint on the next message. That part does not change here. What changes is where those processes live once more than one node can run them.
Where we started
A conversation is a live Elixir process. It holds the full message history in memory, runs its reasoning loop, and checkpoints to Postgres when it goes idle. The next message finds the checkpoint and thaws a fresh one from it.
Before clustering, we ran this on N replicas behind a load balancer. The replicas were isolated: no shared process registry, and no way for one node to see what another was doing. The load balancer had no session affinity, so nothing tied a provider's next request to the node that handled the last one.
It worked anyway, because the traffic was over a WebSocket that pinned the client to a node on join. A provider opens Assist, the browser opens a WebSocket, and that socket stays open to whichever node accepted the connection. Every message on that socket lands on the same node until the socket closes. The load balancer's routing decision only mattered once, at connect time.
We accepted one edge. A provider reloads the page mid-turn, the load balancer sends the new socket to a different replica, and the running state on the old node is invisible until it checkpoints. For a few seconds, the reload sees a stale or empty conversation. We chose N independent, redundant replicas over shared session state, and that edge was the cost. At launch, with no event-triggered flows and reload as the only way to land on a different node, it was a reasonable trade. The only users who reported noticing the flaw were internal users trying to break it.
What changed
We always knew the hole existed. We planned to close it eventually.
Then features still in development started to exercise it on purpose. The first real case was our client-initiated event API, where a flow elsewhere in the product posts an event into a running conversation over plain HTTP. Same client, same conversation, different connection. The load balancer routes that POST per-request, independently of the socket, so it can land on any replica. And it was the front of a class we were already heading toward: background jobs, webhooks, one-off triggers, work that needs to post into a conversation without riding the provider's socket at all.
With N replicas, an event and a socket agree on a node one time in N, so what used to be a rare accident of reload timing became a routine, structural mismatch. No amount of care in the new flows could route around it. Any event-triggered write to a session-scoped conversation hits it, on any node count above one, every time the event and the provider's socket land apart.
The fix could not be scoped to just the new flows. A reload was already exercising the same gap, and the new flows only made it common enough to plan for instead of shrug at.
So the problem split into two properties:
- Any node must be able to show a session. An observer joining from node B needs to see what is happening on node A.
- Only one node may write to a session at a time. Whichever node is running the reasoning loop and appending to the checkpoint must be unambiguous, even while the fleet reshuffles under it.
Why not session affinity?
The obvious idea is to avoid the mismatch entirely by pinning a session's traffic to one node with load-balancer affinity, so the event and the socket always land together.
Most of the future use cases we have in mind have nothing to pin on at all. A background job carries no cookie and no socket. There is nothing to be sticky about.
And where there is something to pin, it still does not hold. The case that came up early already spans two routing paths from one client. The socket is pinned by its connection. The routing decision happened once, at connect, and nothing consults an affinity table again. The event POSTs are pinned by the affinity table, whose cookie entries expire, rebalance on scale events, and re-target when a pod dies. Nothing reconciles the two. If the socket blips and reconnects to a new pod, the cookie still maps to the old one, and every POST goes to the wrong pod, every time, until the table churns again.
Affinity is best-effort placement, and this problem needs a guarantee. We want a session to behave correctly no matter where anything lands. Once that holds, affinity on top is a latency optimization and nothing more.
The easy half and the trap
Clustering plus PubSub answers the observation half. We could connect the nodes, publish agent events to a topic, and then subscribe from wherever the viewer's socket happens to be. That part is well-trodden ground, no need to get into the details on it. The Phoenix Channels guide covers them.
Clustering seems to answer the write half too. If nodes can discover where a session's process is running, and route a turn there instead of starting a second one, what is missing?
What is missing is that discovery is a live, in-memory view, not authority. A process registry is a distributed cache of "who is running what," and it tells you what the cluster believed a moment ago. It is only as good as the moment you ask, and there are moments when it is wrong. A node that just restarted has an empty view. It knows about nothing, including sessions that are alive and well on its peers. A network partition gives two nodes two different views of the same cluster, each internally consistent, each wrong about the other side. Two nodes can each consult their own view, each see no running agent for a session, and each start one. A distributed registry can hold two entries for the same key for a moment while a claim is in flight. None of this is exotic either. It clusters around deploys, restarts, and scaling events, the exact moments a session needs a decision about who owns it.
A view answers "where do things seem to be right now." A view can be eventually consistent. A write, however, needs a durable fact that answers "who is allowed to write right now," one that survives a restart and a partition, and can be checked in one atomic step at the moment of the write. That fact cannot live in the cluster's own transient memory, because the cluster's memory is exactly the thing that goes wrong in the cases above. It has to live somewhere the write itself can check, atomically, every time.
Ownership: the lease
Ownership had to become a durable fact. Where a socket happened to land could not decide it. We store it as one row per session in an agent_leases table: owner node, epoch, expiry. Every read or write against that row is a single atomic statement, evaluated on the database's clock. Whatever the pods' clocks disagree about, the database does not care.
The claim rule reduces to four cases, and one INSERT ... ON CONFLICT statement covers all four.
- No row for this session. Claim it fresh.
- Your own row, not yet expired. Extend it, epoch unchanged, an idempotent re-claim.
- Someone else's row, expired. Take it over and bump the epoch by one.
- Someone else's row, still live. The statement changes nothing, and the caller learns who holds it.
1on_conflict =2 from(l in Lease,3 where: l.owner_node == ^node4 or l.expires_at <= fragment("timezone('UTC', now())"),5 update: [6 set: [7 owner_node: ^node,8 epoch: fragment(9 "CASE WHEN ? <= timezone('UTC', now()) THEN ? + 1 ELSE ? END",10 l.expires_at, l.epoch, l.epoch11 ),12 expires_at: expires_at_after_ttl(ttl)13 ]14 ]15 )1617Repo.insert_all(Lease, source,18 on_conflict: on_conflict,19 conflict_target: [:agent_key])
Every entry point into a session resolves the owner before it does anything else. If this node owns the lease, it runs the turn. If another node owns it, this node forwards the turn there instead of starting its own copy. A provider's request can land on any node; only the owner ever runs the reasoning loop for that session.
The forward is an :erpc call to the owner, and the failure modes are the interesting part. A timeout, a remote exception, or a remote exit all surface to the caller, because the turn may already be running over there. {:erpc, :noconnection} is the subtle one. The erpc contract says the remote body may or may not have been applied, and in one traced run the caller got noconnection back while the remote body kept going for another ten seconds. So that case re-resolves ownership with a fresh claim instead of running the body locally. Postgres is usually still reachable when a peer node is not, and running locally against an owner that is provably alive is a split-brain write, not a fallback. A second unreachable hop fails the turn.
If the lease store itself is unreachable, the node refuses the turn rather than run it unfenced. Every failure mode ends in a refusal or a re-resolution, and none of them degrades into a local write.
Reads take the opposite trade. A rejoin asking what is in flight uses a short timeout and falls back to a safe default, because a lost read blanks a view for a moment and never corrupts anything.
A supervised heartbeat renews the lease on its own timer, outside any process that could block. A reasoning loop stalled on a slow tool call or a wedged model response does not stall it. The heartbeat tracks the session's lifecycle. It renews for as long as the agent lives, and it stops when the agent idles out and shuts down.
The fence
A node can expire out of its lease while its process is still alive and about to write a checkpoint. No amount of TTL tuning removes that window. It only moves it. That is why the lease needs a second mechanism behind it.
The fence makes that window harmless instead of trying to eliminate it. It enforces the epoch at the one place a write can land.
A lease with no fence trusts clocks and process liveness to line up. They do not always line up. A fence with no lease never corrupts anything, but it has no coordination behind it, so every node with an opinion fights over every write with no shared answer for who should even be trying.
The mechanism is a conditional upsert. Every checkpoint write carries the writer's epoch. The upsert only lands if the stored epoch is not ahead of the writer's own. A stale write changes zero rows, and the writer finds out in the same statement that told it to write.
1on_conflict =2 from(c in Checkpoint,3 where: is_nil(c.epoch) or c.epoch <= fragment("EXCLUDED.epoch"),4 update: [5 set: [data: fragment("EXCLUDED.data"),6 epoch: fragment("EXCLUDED.epoch")]7 ]8 )910case Repo.insert_all(Checkpoint, [row],11 on_conflict: on_conflict,12 conflict_target: :key) do13 {count, _} when count > 0 -> :ok14 {0, _} -> {:error, :fenced}15end
A rejected write means this node no longer owns the session, whatever it believed a moment ago. There is nothing to retry.
1case Storage.put_checkpoint(key, data, opts) do2 :ok ->3 :ok45 {:error, :fenced} ->6 halt_turn_and_stand_down(agent_key)7end
The losing node halts the turn where it stands. It never broadcasts the partial result it was about to send, and it stands the session down rather than trying again with the same stale epoch.
From the provider's side, the stream stops. The answer that was in flight is gone. The next message re-claims the session on a healthy node and thaws from the last checkpoint, so the conversation survives intact. That one turn does not.
The fence is hard to observe from outside, so we test it directly. A cluster canary starts two real nodes against a real database, has the second claim a session the first still believes it owns, and checks that the first node's next write comes back rejected. In production it never has, which is what we want from a backstop, and it is why the canary carries the proof instead.
Takeover and recovery
A pod can die without warning: killed, crashed, evicted. Its lease keeps ticking down on its own until the TTL passes, and the next message for that session claims ownership on whichever node receives it. The epoch advances by one, and the new owner thaws the session from its last checkpoint. The provider never has to know which pod they were talking to.
A node that receives a message inside that window refuses it instead of running it in the wrong place, and the refusal never reaches the provider as an error. The server answers with a retry directive carrying a delay and a budget, and the client re-sends inside it. The provider sees a slightly longer wait. If no directive comes back, the client surfaces the failure instead of inventing a retry of its own.
The numbers are small on purpose: a 15-second TTL, a 5-second heartbeat, three renewals inside every lease window. Kubernetes' own leader election is at the same scale, a 15-second lease renewed every 2. The exact numbers differ. The shape is the common part. The beat is several times shorter than the window, so a missed renewal or two costs nothing. The window is short enough that a dead node does not hold a session long.
A deploy is the case a pod knows about in advance. A draining pod hands its leases back and closes its channels, and the sessions it was holding shut down within the drain window rather than waiting for a TTL to lapse. Because the handoff happens ahead of the shutdown signal, a rolling deploy costs providers no waiting for the session to come back. The next message for a handed-back session finds an already-claimable lease instead of an expired one.
Observing from anywhere
This is the half that pays off for a provider directly. Agent events publish to a topic scoped to the session, and whichever node a provider's socket is connected to subscribes to that topic and streams the events through. The node running the turn and the node showing it to the provider are often not the same one. Neither needs to know where the other runs.
Joining mid-turn works the same way a live join always should. A short replay fills in whatever happened before the socket connected, and the live stream picks up from there. A provider who refreshes the page, or opens the same conversation in a second tab, sees the turn in progress exactly as if they had been watching the whole time.
This is the picture from earlier, with the outcome fixed. The event still starts work on whichever node it lands on. The provider's socket still connects to whichever node the load balancer sends it to. The two no longer need to be the same node. The lease settles the write side, and PubSub and replay settle the observe side.
This is what closes that edge for good: refreshing the page out of habit, or coming back to something that was already running. Those moments were rare, and most providers never hit one, but when they hit, the conversation looked stale or empty until the next checkpoint landed. The event-triggered features would have turned that from rare to routine. On isolated nodes, whether a provider saw a conversation start or nothing at all was a dice roll on where the event landed. Now the provider sees the turn that is running, every time.
Prior art and what comes next
None of this is a new idea, and we did not set out to invent one. Orleans models this as virtual actor placement with a single active instance per identity. Akka Cluster Sharding uses a lease as a backstop against split-brain shard ownership during a partition. Kafka fences producers with an epoch so an old, zombie producer instance cannot write after a newer one has taken over. Kubernetes leader election is a lease with a TTL and a retry interval, the same shape we arrived at ourselves. Different systems, solving different problems, keep landing on the same pair: a lease for who is allowed, and a fence for whose writes stick. We converged on it too, from our own constraints.
Single-writer ownership makes concurrent sessions safe. It does not yet make a single run durable. If a pod crashes or a rolling deploy lands mid-turn, the in-flight answer can still be lost; the next owner thaws from the last checkpoint, not from the middle of the turn that was running. The next problem is recovering a run in progress. We want to continue from where it stopped, not from where it last saved. That is what we are thinking about now.
