Memory in the loop.

abstract

Language agents run a loop of observing, reasoning, and acting, but the memory they reason over is treated as something outside that loop: a store queried at most once per turn. We study the regime in which memory moves inside the loop, read and written on every reasoning step. The standing objection is latency: networked vector stores answer in tens to hundreds of milliseconds. That cost is a property of where the store lives, not of the pattern; an in-process store answers in ~100 µs, and at that speed the per-step tax collapses. By the extended-mind parity criteria, a store fast enough to be transparent to a reasoning step, in a loop wired to consult it, functions as extended working memory. The claim is causal, not correlational: holding the per-turn memory budget fixed and varying only the store's answer speed, redundant actions rise monotonically from 0.0 of 12 at in-process speed to 7.2 of 12 at a 110 ms round trip (two models, exact permutation p = 0.0079). End to end, a bounded context window drives recall to 0/5 across four GPT-5-class models; in-loop memory recovers 3.6–4.8/5 with its misses all traced to the agent's read policy, never the store, and a write-side dedup gate, fixed in advance from that diagnosis, recovers 4.8–5.0/5. An instructed restate-every-reply baseline also solves the task, but pays per-turn rent: by turn 25 the gated store is cheaper on every model at list prices, at equal judged accuracy. The remaining bottleneck is network embedding (~200–400 ms); a small local embedder returns the complete operation to a measured ~40 µs.

what it shows

110 ms → 165 µs

A cloud vector store answers in about 110 milliseconds per round trip. The in-process store's live ops measured 80–165 µs at the median, three orders of magnitude closer.

0.36 vs 953

Checks affordable per reasoning step, at a 10% latency budget and one second of reasoning: a cloud store plus network embedder (complete check ≈ 312 ms) affords 0.36; an in-process store plus local embedder (≈ 116 µs) affords about 953.

0.0 → 7.2 of 12

Hold everything fixed and vary only the store's answer speed: redundant actions rise monotonically with latency, from zero at in-process speed to 7.2 of 12 at the cloud round trip.

500 ms → 1.6

Five times the per-turn budget still leaks 1.6 of 12. The round trip is the problem, not the allowance.

244 / 244

The store never lost a fact in any run. Every miss traces to the agent's read policy, never the store.

0/5 → 4.8–5.0

A bounded window drops recall to 0/5. In-loop memory recovers 3.6–4.8/5; a write-side dedup gate, its threshold fixed in advance from the diagnosed miss, closes it to 4.8–5.0/5 and blocked 109 duplicate saves with zero errors.

the dose-response

Hold the per-turn budget fixed at 100 ms and inject latency into the store alone. The guard code is identical in every arm; only the answer speed moves.

the budget sweep

Raising the allowance does not buy the guard back. At the 110 ms rung, every affordable lookup costs a full round trip of turn time, so the buyback is sublinear: even half a second of memory dead time per turn still leaks 1.6 of 12.

the per-step tax

Let a turn run twenty in-loop memory checks (S = 20). The same twenty, priced by where the store lives:

retrieval blocking time per turn, S = 20 in-loop checks · store op ≈ 85 µs measured, network adds only the round trip · Table 2 of the paper

measured, not modeled

Even with no network at all — a real Qdrant container on the same machine over loopback — the client-server boundary costs about 4.2× at the median on this host. Adding a realistic 110 ms WAN to the measured op reproduces the modeled 2.20 s per turn almost exactly.

per-op recall latency · 1,000 vectors, 2,000 trials, one host · Table 3 of the paper

end to end

Under a bounded window, every baseline run scores 0 of 5, and telling the model its window is short changes nothing. Restating every constraint in every reply solves this five-fact task perfectly, at a token rent that grows with the working set; in-loop memory recovers 3.6–4.8 of 5 with a store that never lost a fact. Every miss was read-policy, not lost data, so a write-side dedup gate, its threshold fixed in advance from that diagnosis, closes the gap to 4.8–5.0 of 5.

constraints recalled (of 5), mean [min–max] over five repeats per cell · memory+gate adds a silent write-side dedup check, its threshold fixed in advance from the diagnosed miss · a second, rubric-based grader matches every cell · Table 1 of the paper

the cost axis

Restating every constraint in every reply also solves the task, so accuracy alone does not separate the two. Cost does. Stretched to 25 turns, all three conditions still land at 5/5, but the bills diverge: restatement pays flat-rate rent in expensive output tokens on a bill that grows every turn, while the gated store pays for its writes early and then tracks baseline, reading a fixed eight lines however large the store grows. The curves cross by turn 12; by turn 25 the store is cheaper on every model.

The store wins on the price sheet, not just the total: its spend is cheap, cacheable input tokens, while restatement's is output text priced at six to eight times the input rate and never cache-discounted. At turn 25, restatement runs 1.23–1.58× the baseline bill across the four models; the gated store runs 1.00–1.26×, undercutting restatement on every one and reaching parity with baseline itself on the smallest model.

cumulative cost at turn 25 as a multiple of the baseline bill · five repeats per cell · list prices · token-cost figure of the paper

cite

@misc{khan2026memoryintheloop,
  title={Memory in the Loop: In-Process Retrieval as
         Extended Working Memory for Language Agents},
  author={Khan, Yusuf and Lipizzi, Carlo},
  year={2026},
  eprint={2607.05690},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2607.05690}
}