We measured our own workspace — token economics (method + results)

The setup

20 realistic questions an agent asks this workspace. For each, two ways to answer — and we count the tokens each returns into the agent's context (tokens ≈ chars/4, no tokenizer dependency):

• (a) Index discipline — search (semantic), then read the one document that answers (plus a one-time orientation read of the index + system map + latest log, amortized across the session).
• (b) Undisciplined — search, then read every hit in full, no snippet triage.

Correctness is held equal by construction: (a) reads the doc that actually answers; (b) reads that same doc and the rest. Measurement is deterministic — it's byte accounting over the real API responses, not a model run.

Results — 15 of 20 tasks

These are the tasks where semantic search surfaced the right doc (see "where it broke" for the other 5). Sorted by savings.

Headline: ~3.6k tokens per task vs ~21.4k — 5.94× fewer at the margin, 4.98× once you load the one-time orientation cost onto just these 15 tasks (it amortizes further over a longer session). For reference, loading the entire workspace is ~193k tokens — the ceiling index discipline avoids.

Where it failed

Two things we found by looking, and are fixing. They matter more than the multiple.

1. Semantic search works; our default (keyword) doesn't — and the default is what agents get. With the default keyword mode, the correct document surfaced in only 6 of 20 questions — plain-English, multi-word queries returned zero hits, or a log file that merely mentioned the word. Switch to semantic: 15 of 20 (18/20 with better wording), 13 of them at rank 1. So we're making semantic the default. The savings are real, but they ride on retrieval actually finding the doc.

2. Two of our own documents were invisible to search. Even with strong on-topic queries, two canonical docs never surfaced semantically — their embeddings were never computed. We only found it because we went looking. (Fix: audit embedding coverage.)

Neither is hidden. That's the point: memory legible enough to measure — and honest enough to tell you when it's wrong.

Re-run it yourself

The harness is ~60 lines. Point it at your own workspace with an API key. Fill TASKS with real questions and the doc that answers each (the "winner"); it reports the multiple.

// bench.mjs — node bench.mjs  (needs SYNCPEN_API_KEY in env; Node 18+)
const BASE = "https://www.syncpen.io/api/mcp";
const H = { Authorization: `Bearer ${process.env.SYNCPEN_API_KEY}`, "Content-Type": "application/json" };
const tok = (c) => Math.round(c / 4); // tokens ≈ chars/4

async function get(path, params) {
  const url = new URL(path.replace(/^\//, ""), BASE + "/");
  if (params) for (const [k, v] of Object.entries(params)) if (v != null) url.searchParams.set(k, String(v));
  const r = await fetch(url); const t = await r.text();
  if (!r.ok) throw new Error(`${r.status} ${path}`);
  return { json: JSON.parse(t), bytes: t.length };
}
const size = new Map();
const sizeOf = async (id) => size.has(id) ? size.get(id)
  : size.set(id, (await get(`/documents/${id}`)).bytes).get(id);

// Your questions: a semantic query + the doc id that correctly answers it.
const TASKS = [
  { q: "how does billing work", answer: "cmXXXXXXXXXXXXXXXXXXXXXXXX" },
  // ...20 of these
];

let sumA = 0, sumB = 0;
for (const t of TASKS) {
  const s = await get("/search", { query: t.q, limit: 8, mode: "semantic" });
  const hits = (s.json.results || []).map((h) => h.id);
  if (!hits.includes(t.answer)) { console.log("MISS (query too weak):", t.q); continue; }
  let b = s.bytes; for (const id of hits) b += await sizeOf(id);   // read every hit
  const a = s.bytes + (await sizeOf(t.answer));                    // read the winner only
  sumA += a; sumB += b;
  console.log(t.q, "→", tok(a), "vs", tok(b), "tokens", (b / a).toFixed(1) + "×");
}
console.log(`\nTOTAL: ${tok(sumA)} vs ${tok(sumB)} tokens — ${(sumB / sumA).toFixed(2)}× fewer`);

Notes: mode: "semantic" matters (finding #1). "Winner not in hits" means the query was too weak — refine it, the way an agent would. chars/4 is a proxy, stated openly. This measures the retrieval footprint — the claim — not end-to-end correctness, which we held equal by design.