Every test we ran

What was tested

Whether Tideline's memory lowers the bill of an agent answering questions about a codebase

Dataset

176 questions from SWE-QA, a public code-question benchmark, that no earlier agent conversation had used; 15 open-source repositories (flask, astropy, sympy and others), 9,576 files, 43,647 stored passages

Setup

A read-only agent with 3 tools, capped at 12 model calls and 2,048 output tokens per call. Claude Sonnet 5 through OpenRouter, provider pinned to Anthropic. Each question asked once with memory on and once off. Two judges: GPT-4o and Gemini 2.5 Pro.

Also measured

Model calls per conversation down 24.8%. Runs cut off at the 12-call limit: 15.3% → 5.1% (27 → 9 of 176). Billed tokens down 26.9% (57,984 → 42,404 per conversation). Runs that ended with an answer, right or wrong: 71.0% → 84.7%. Correct answers: 82.4% → 88.1% (first judge) and 76.7% → 85.2% (second judge); the first judge's range reaches zero, so this is no claimed gain. On the 120 questions both versions answered: −4.2 and 0.0 points.

Pre-registered

Yes, independently timestamped, 23 September 2026

Record

2026-09-23-heldout-results.md

Records and scripts available to your engineer on request.

What was tested

The A1 design on five vendors' models

Dataset

The same 176 questions

Setup

As A1, one pre-registration per model. Each model on its own provider.

By model

Claude Sonnet 5: bill down 16.3%, tokens 57,984 → 42,404. Kimi K2: bill down 18.4% (range 8.4% to 27.8%), tokens 46,892 → 38,140; correct answers lean lower, not significantly. Qwen3-Coder: bill down 29.7% (range 18.2% to 40.2%), tokens 43,018 → 29,869; 7–9 points fewer correct answers where both versions answered. Gemini 2.5 Pro: down 5.2%, not significant. GPT-5.4: up 0.5%, no effect.

Why the models differ

Memory saves where an agent explores before answering. GPT-5.4 and Gemini 2.5 Pro answer these questions in 3 to 4 calls, which leaves little to save.

Pre-registered

Yes, 23 September 2026

Record

2026-09-23-heldout-models-results.md

Records and scripts available to your engineer on request.

What was tested

The same design on the questions the build was developed on

Dataset

540 SWE-QA questions, sampled by a pre-registered rule from a public randomness beacon

Setup

As A1; 1,620 conversations per run

Also measured

Model calls down 24%. Correct answers 80.2% → 88.5% under one judge, registered as a guardrail for the cost claim.

Note

The two runs share one harness and one question set; they are two runs of one test

Record

2026-09-22-m3-final-proof.md, 2026-09-23-m3-replication.md

Records and scripts available to your engineer on request.

What was tested

Tideline's new recall ranking (BM25, which weights each word by how rare it is among what the asker may read) against its previous ranking, with memory on in both

Dataset

The 540 in-sample questions; 456 of them name a file the check can score

Setup

As A1, Claude Sonnet 5. The ranking uses no AI model.

Also measured

Bill down a further 6.4% (range 2.8% to 9.9%) against the previous ranking. Tokens down 12.7%. Correct answers +1.1 and +1.9 points, within the margin. Recall search time 154 ms → 54 ms median, 256 → 117 ms at the 95th percentile, after a change made following the run and checked to return the same results.

Note

Measured against our previous ranking; it is never added to A1's figure

Pre-registered

Yes, 24 September 2026 (search time: no)

Record

2026-09-24-recall-bm25-results.md

Records and scripts available to your engineer on request.

What was tested

BM25 ranking, pointers to other likely files, and an answer-time evidence check, with memory on against off

Dataset

716 code questions for Claude Sonnet 5; 176 for Qwen3-Coder and Kimi K2. All of them informed the design.

Setup

As A1; 2,136 conversations

Also measured

Runs cut off at the limit 12.3% → 4.9%. Cost per correct answer down 15.4% and 14.9%. Tokens down 21.0%. Qwen3-Coder bill down 21.9%; Kimi K2 down 21.1%.

Pre-registered

Yes, 25 September 2026

Record

2026-09-25-directions-gate-results.md

Records and scripts available to your engineer on request.

What was tested

BM25 recall ranking with memory on against off, before pointers and the evidence check were added

Dataset

714 of 716 code questions for Claude Sonnet 5; 176 for Qwen3-Coder and Kimi K2

Setup

As A1

Also measured

Qwen3-Coder bill down 44.0% (range 34.6% to 52.9%); Kimi K2 down 34.8% (range 25.1% to 44.1%). Claude answers right overall +5.7 and +3.2 points.

Limits

Where both versions answered, Claude was judged right 1.9 and 4.1 points less often, both ranges below zero. Kimi K2 lost 9.2 points where both answered, under one judge.

Pre-registered

Yes, 24 September 2026

Record

2026-09-24-bm25-on-off-results.md

Records and scripts available to your engineer on request.

What was tested

Whether choosing, per memory block, between quoting a passage and sending a one-line pointer lowers the bill further

Dataset

176 held-out code questions; 1,408 conversations per model

Setup

Against A5's configuration, on Claude Sonnet 5, Qwen3-Coder and Kimi K2

Also measured

On tasks given pointers, agents made more model calls (Claude +0.57 per conversation), which cancelled the saving the offline estimate predicted

Pre-registered

Yes, 27 September 2026

Record

2026-09-26-w1-0-paired-run-results.md

Records and scripts available to your engineer on request.

What was tested

Whether telling the model the memory block "may be incomplete or not about this question" stops it answering too early

Dataset

540 in-sample code questions

Setup

Qwen3-Coder, the new wording against today's, memory on in both

Also measured

Model calls per conversation up 7.5%

Pre-registered

Yes, 23 September 2026

Record

2026-09-23-recall-wording-results.md

Records and scripts available to your engineer on request.

What was tested

How well Tideline's recall answers questions about long chat histories

Dataset

LongMemEval-S, 500 questions, each with its own history of about 113,000 tokens; 6 question types (single-session user, assistant and preference; multi-session; temporal reasoning; knowledge update). Stored as 579,354 passages in 500 separate tenants.

Setup

One neutral answer prompt for every version; every failed question counted wrong; graded by the benchmark's own GPT-4o grader. Readers: GPT-4o and Claude Sonnet 5 (thinking off). Recall at its default budget and at a limit of 20 passages.

Also measured

GPT-4o at the default budget: 55.8% (range 51.4% to 60.1%), about 889 tokens per question. Claude Sonnet 5: 59.2% at the default budget and 67.4% at limit 20. No memory: 8.8% (GPT-4o).

Note

The limit of 20 was chosen after a retrieval check on the same 500 questions. The grader also reads the GPT-4o answers.

Pre-registered

Yes, 27 September 2026

Record

2026-09-27-w7-12-lme-results.md

Records and scripts available to your engineer on request.

What was tested

Tideline's recall against sending the whole history to the same model

Dataset

293 of the 500 LongMemEval-S questions (every question of three types, the first 20 of each of the other three)

Setup

One reader throughout, GPT-4.1 at temperature 0; the same answer prompt; the benchmark's own grading

Also measured

Correct: 54.6% with Tideline at the default budget (about 893 tokens per question), 63.8% at limit 20 (about 1,552 tokens), 81.6% with the whole history (about 113,000 tokens). Against a history the provider has cached, the cost ratio is 4.5 to 10.5×, computed. Tideline sent 99% fewer prompt tokens.

Note

A trade: fewer tokens and a far lower cost, at lower accuracy. In the earlier run of this comparison (22 September), answers were at parity where one recalled fact answers the question (knowledge update 78% against 80%), and the loss came where an answer must be assembled from several sessions (multi-session 30% against 68%).

Pre-registered

The Tideline runs yes (27 September 2026); the whole-history run (22 September) no

Record

2026-09-27-w7-12b-lme-results.md, 2026-09-22-lme-full-context.md

Records and scripts available to your engineer on request.

What was tested

How well Tideline's recall answers questions about long multi-session conversations

Dataset

LoCoMo, 10 conversations, 1,540 questions (categories 1–4); 2,143 stored passages

Setup

One neutral answer prompt; every failed question counted wrong; LLM-judge accuracy (GPT-4o-mini judge, plus a second judge from another model family). Readers: GPT-4o-mini and Claude Sonnet 5 (thinking off).

Also measured

Claude Sonnet 5 at the default budget: 74.3%. GPT-4o-mini: 67.1% at the default budget (about 950 tokens per question), 73.2% at limit 20, 80.4% with the whole conversation (about 22,575 tokens), 24.7% with no memory. The second judge reached the same decisions. With no cache, the whole conversation costs 7–19× Tideline per correct answer.

Note

The whole conversation is more accurate on both readers. Against a warm cache, Tideline at limit 20 on Claude costs more per correct answer.

Pre-registered

Yes, 27 September 2026

Record

2026-09-27-w7-12c-locomo-results.md

Records and scripts available to your engineer on request.

What was tested

How often recall puts the right source first using only word statistics, against small embedding models

Dataset

Code questions: 144 held out and 456 in sample, scored against the file the answer names. General documents: NanoBEIR, 649 questions across 13 public retrieval datasets (HotpotQA, NQ, MS MARCO, FEVER and others).

Setup

Offline; no paid call; no AI model in the shipped ranking

Also measured

General documents: first result relevant 41.4% → 50.1% with no model; a 33M-parameter model added on top reached 60.6%. Code questions with one or two typos: 49.1% and 48.6%. An earlier check on 21 September, before BM25: the recall block held a file the answer names for 39.2% of 531 questions.

Pre-registered

No

Record

2026-09-24-retrieval-bake-off.md, 2026-09-21-retrieval-quality-census.md

Records and scripts available to your engineer on request.

What was tested

Leaks across users and companies, withdrawal, revocation, updates and the audit record, each beside a check that could have failed

Dataset

A synthetic company corpus of about 1,000 documents with 60 planted secrets; 20,760 requests; 120 open conversations

Setup

Tideline's gateway against a recording stand-in for the AI provider; Claude Sonnet 5 for the live withdrawal check

Also measured

Control: the permitted user received the secret 2,181 of 3,000 times. Withdrawal: 0 of 720 later requests carried a withdrawn document (it rode 840 of 840 in untouched twin conversations); 0 of 32 on Claude Sonnet 5. Revocation: 0 of 120. Updates: 0 of 480 new requests got an outdated version. Audit: all 68,624 hand-overs recorded. Cost: +16 ms per new recall (range 15–16 ms).

Limits

A count, not a rate. The model repeated a withdrawn fact it had already seen in 8 of 16 conversations. Conversations already open keep an updated document's old version (80 of 80). A withdrawal re-writes the cache once: +17% on a five-turn conversation's bill.

Pre-registered

No. Run on 23 September and re-run on 26 September 2026; every result held.

Record

2026-09-23-governance.md, 2026-09-26-tenancy-rerun-results.md

Records and scripts available to your engineer on request.

What was tested

Usefulness, access control and forgetting, scored together by a third-party benchmark

Dataset

GateMem's office domain: 17 episodes, 547 checkpoints (154 usefulness, 171 access control, 222 forgetting); synthetic data

Setup

Seven configurations; reader Claude Sonnet 5; judges GPT-4o and Gemini 2.5 Pro

Also measured

The controls kept restricted text out of the AI's input in 135 of 171 access-control checks (2 of 171 with them off). Lowest forgetting leakage of any configuration (0.5%). Cost 0.275× long context per checkpoint, at 18.2 points less usefulness.

Note

The checkpoints informed our adapter's rules. The held-out domains have not run yet.

Pre-registered

Yes, 25 September 2026

Record

2026-09-25-gatemem-results.md

Records and scripts available to your engineer on request.

What was tested

Whether memory added to a request makes the provider lose its cached copy of the conversation, which costs about ten times more to re-send

Dataset

11,884 consecutive request pairs in four runs of the A1/A3 agent (3,628, 4,100 and 3,380 in sample; 776 held out)

Setup

Claude Sonnet 5; conversations in which nothing was withdrawn

Note

An earlier build lost the cache on 770 of 3,448 pairs; that defect was fixed and re-measured. On GPT-5.4, whose provider caches on a best-effort basis, both versions lose the cache now and then.

Pre-registered

The held-out run yes; the three in-sample runs no

Record

2026-09-23-m3-replication.md, 2026-09-23-heldout-results.md

Records and scripts available to your engineer on request.

What was tested

Whether an instruction planted in a stored document is obeyed when memory hands that document to an AI

Setup

A harmless planted instruction inside Tideline's labelled memory block; two popular models

Note

Tideline controls who may write to memory and who may read from it. It does not stop a model obeying text inside a document.

Record

2026-09-23-governance.md

Records and scripts available to your engineer on request.

What was tested

What Tideline's credential masking changes in the text agents read, measured before it shipped

Dataset

The 43,647 passages of the A1 corpus (15 repositories, 9,576 files)

Also measured

The first version of the rules changed 73 passages, most of them ordinary code such as password = self.cleaned_data.get(...); it was hardened before shipping

Pre-registered

No, 25 September 2026

Record

2026-09-25-release-redaction.md

Records and scripts available to your engineer on request.

What was tested

Whether any of Tideline's size-reducing steps ever makes a request larger on the wire

Dataset

139 public command and result pairs from three open-source repositories, checked on every build; 138,217 pairs from real agent sessions on a developer machine

Also measured

Deleting or loosening each safeguard makes the build fail, which shows the check catches a regression

Pre-registered

No; it runs in the test suite on every build

Record

2026-09-25-never-worse-history.md

Records and scripts available to your engineer on request.

What was tested

Claude Code doing coding work with Tideline's memory, through its plugin and through its gateway

Dataset

The 17 tasks of THOL, a public leaderboard for tools that cut a coding agent's bill

By setup

Plugin on a Claude subscription (Claude Opus 5.5, Claude Code 2.1.301, 102 sessions, 21 October 2026, pre-registered): 1.01 of vanilla Claude Code (range 0.95 to 1.07). Plugin, earlier build (Claude Sonnet 4.6, Claude Code 2.1.234, 170 sessions, not pre-registered): 3.4% lower (range 11.5% lower to 5.8% higher). Gateway against Claude Code talking to the provider directly (same model and client, 255 sessions, not pre-registered): 7.7% higher (range 0.2% lower to 16.9% higher). Memory on vs off with both versions behind the gateway: 6.9% lower across the 17 tasks (range 1.3% to 12.1% lower), 9.1% lower on the 13 tasks where memory had anything to hand over, and 1.004 on the 4 tasks with nothing to recall, which act as a placebo. The gateway itself adds 14.7%, so the net against direct use is the 7.7% above.

Note

Coding agents already read the files memory would hand them. The September runs use Claude Code's own cost figure.

Record

2026-10-21-plugin-on-subscription/RESULT-S1.md, 2026-09-24-plugin-vs-vanilla-claude-code.md, 2026-09-23-coding-leaderboard-three-groups.md

Records and scripts available to your engineer on request.

What was tested

Claude Code searching folders of files, with and without Tideline's plugin

Dataset

xAFS, 110 questions over 13 folders of 5 to 9,988 files; 390,252 stored passages; CC-BY-4.0

Setup

Claude Code 2.1.282, Claude Sonnet 5, OpenRouter to Anthropic; 220 sessions; two judges

Also measured

Correct: 77.3% against 72.7% where both answered, within the margin. On the two largest folders, memory did not reach the agent in time.

Pre-registered

Yes, 27 September 2026

Record

2026-09-27-w7-14-afs-results.md

Records and scripts available to your engineer on request.

What was tested

Claude Code implementing feature tickets with memory on and off

Dataset

12 one-line feature tickets (6 frontend, 6 backend) on a public FastAPI and React template

Setup

Headless Claude Code 2.1.234, Claude Haiku 4.5, 4 runs per ticket and version, both versions through the same gateway

Note

A first run showed memory making the agent over-build; the fix (pointing at a repository's documents instead of quoting them) gave this result

Pre-registered

No, 22 September 2026

Record

2026-09-22-public-agentic-benchmark.md

Records and scripts available to your engineer on request.

What was tested

How much Tideline's lossless compression of shell output removes from real agent sessions

Dataset

1,999 real coding sessions

Also measured

A compressed output shrinks 18.6% at the median; most output is not safely compressible

Note

Tideline's bill saving comes from fewer model calls; compression contributes little

Pre-registered

No, 22 September 2026

Record

2026-09-22-headline-numbers.md, 2026-09-26-fold-reach.md

Records and scripts available to your engineer on request.

What was tested

The time Tideline adds to a request

Also measured

Governance adds 16 ms per new recall, one request at a time on about 1,000 documents. A recall search takes 54 ms median on 43,647 stored passages.

Note

Search time grows with the size of the store; latency under heavy concurrent use is not yet measured

Record

2026-09-23-heldout-results.md, 2026-09-23-governance.md, 2026-09-24-recall-bm25-results.md

Records and scripts available to your engineer on request.

What was tested

Whether one company's burst of writes slows another company on the same server

Setup

Fresh deployments of the build before and after the fix, each run twice in alternation

Also measured

Ingest latency during a burst: +271 ms → +5.9 ms. New recalls at 64 in flight: about 82–90 → 231–236 a second. Bulk ingest of 579,354 sources at 137 a second.

Pre-registered

Analysis plan committed before the runs, 26 September 2026

Record

2026-09-26-tenancy-rerun-results.md

Records and scripts available to your engineer on request.

What was tested

How recall time grows with the size of a company's store

Dataset

13 stores from 59 to 212,889 passages

Limits

On a memory-short test machine, lookups missed the Claude Code plugin's 2-second wait from about 19,000 passages and never reached the agent from about 99,000. E6 is the fix.

Pre-registered

No, 28 September 2026

Record

2026-09-28-recall-scale-profile-results.md

Records and scripts available to your engineer on request.

What was tested

Whether narrowing a key to a few areas of memory slows the statements it runs

Dataset

A restored copy of the A1 store (43,647 passages)

Pre-registered

No, 27 September 2026

Record

2026-09-27-key-scopes-performance-m3.md

Records and scripts available to your engineer on request.

What was tested

Adding people and groups to Tideline's access model, against the build before it

Dataset

Stores of 10,000, 30,000 and 100,000 passages; 600 query pairs at each size; up to 10,000 people

Also measured

Resolving what a person may read, at 10,000 people: 0.44 ms median; a heavy profile 10.02 ms at the 95th percentile, against a 25 ms target

Limits

At the target shape of 200 channels of 50 people, the audience check refused two common, legitimate cases. It refuses safely, so nothing leaks; plan 2 addresses it.

Pre-registered

No, 5 October 2026

Record

2026-10-05-org-brain-baseline.md

Records and scripts available to your engineer on request.

What was tested

Tideline's governed search for one company of 1,000 people, from 10,000 to 1 million passages

Setup

13 Google Cloud c3-standard-22 machines; PostgreSQL 17 and 14

Also measured

Ingest 18.8 ms per passage against 28.0 ms for plan 1, 33% faster. A typical user's search at the 95th percentile: 75 ms at 10,000 passages, 108 ms at 30,000, 300–345 ms at 100,000, 727–757 ms at 300,000, 2.81–2.88 s at 1 million. Access check worst case 28.6 ms, against a 50 ms target. The tests caught 478 of 478 planted bugs.

Limits

A heavy user (50 to 200 containers) misses the target at every size: 405 ms at 10,000 passages. Withdrawing 10,000 items takes 5.8–6.0 s, against a 1 s target. Three causes of the slow search are diagnosed and their fixes planned.

Pre-registered

No, 11 October 2026

Record

The plan 2 exit ledger and findings (not yet in docs/tideline-measurements/)

Records and scripts available to your engineer on request.

What was tested

Tideline v1 on the datasets of agentmemoryleaderboard.ai, run locally on the leaderboard's protocol

Setup

The leaderboard's Add and Search interface and its required reader, GPT-4o-mini; 4–5 September 2026

By dataset

LoCoMo-Refined 61.14% (no-memory floor 7.45%). LongMemEval-S 60.60% on all 500 (floor 12.80%). BEAM 35.77 on all 2,000 questions, 10M included (range 33.91 to 37.60; floor 18.06). PersonaMem-v2 41.4% on 5,000 (floor 33.2%). ATM-Bench 42.12% at k=3 and 31.23% at k=100 (floor 2.27%). Mem-Gallery F1 0.5374 on 1,711 (floor 0.1892). CL-bench under the GPT-4o-mini judge: 14.50% strict and 38.61% rubric on 0–4k contexts (662 tasks), 4.59% and 30.54% on 16–32k (196 tasks). LoCoMo-Refined is scored by the leaderboard's Qwen3-14B judge; GPT-4o-mini gave 71.85%.

Limits

No overall rank can be computed from the leaderboard's published data. Against the top-15 cutoffs: governance 33.72 against 33.80, inside the noise; personalization 53.21, rank 35; safety 31.28 against 42.58; execution about 22, rank 57–59 of 66. Two of the leaderboard's datasets could not be run: ScriptMem (its data for 363 of 457 questions was never released) and LongMemEval-Refined (held out by the leaderboard).

Note

The campaign's report lives outside this repository; check each figure against it before publishing

Pre-registered

No

Records and scripts available to your engineer on request.

What was tested

Tideline's working-conventions ruleset (version 2) in Claude Code, on against off

Dataset

The 17 THOL coding tasks, 15 repetitions, 2 versions: 510 sessions

Setup

Claude Code 2.1.301, Claude Opus 5.5, a Claude Max subscription login, the AgentPark plugin

Also measured

Output tokens 0.829; wall time 0.867; model requests 0.904

Note

Usage on a subscription, not money paid; a separate feature from memory, never added to a memory figure

Pre-registered

Yes, 22 October 2026

Record

2026-10-22-ruleset-ab/RESULT.md

Records and scripts available to your engineer on request.

What was tested

Tideline's gateway giving the fixed part of Claude Code's system prompt its own cache entry

Dataset

22 pairs of new sessions per run, a fixed one-word task, each session's first two requests

Setup

Claude Code 2.1.282 in print mode, OpenRouter's route to Anthropic

Also measured

$0.0088 (21.7%) under Claude Code's own model id; $0.0035 (9.9%) on Claude Opus 5.5

Note

A saving per new session; smaller on longer sessions

Pre-registered

Yes, 26 September 2026, three runs

Record

2026-09-26-w2-4-r1-static-split-bill-results.md and the R1b and R1c records

Records and scripts available to your engineer on request.

What was tested

Tideline's gateway switching on the provider's prompt caching for an app that does not use it

Dataset

One designed 12-turn conversation per version

Setup

Claude Haiku 4.5, provider-billed, turns 26–30 s apart

Also measured

38.6% on a 4-turn conversation; 18% higher on a 3-turn conversation with turns more than five minutes apart

Note

One conversation per version, so no range. It does not apply to Claude Code, which already caches.

Pre-registered

No, 21 and 23 September 2026

Record

2026-09-22-headline-numbers.md

Records and scripts available to your engineer on request.