Digital ScribeA journal by Damian Chetty
← All writing
6 min read

The Knowledge Base as Instrument

How I feed context to coding agents — and what the research says about my choices.

The setup

Here is the arrangement I work with: a Git repository holding everything I need a model to know. Six classes of knowledge — domain, code reviews, code snippets, templates, artifacts, workflows. All YAML. Nothing goes to the model by default. For each task, I select what's needed and inject it.

When I started, this felt like a personal quirk. It doesn't anymore. Parts of the field have converged on a similar shape, the research behind that shape is more interesting than the hype around it, and the failure modes are predictable enough to name in advance.

Let me walk through it.

The convergence I didn't plan

Two empirical studies offer a useful view of convention files — the AGENTS.md and CLAUDE.md files that have become the default interface between repos and coding agents. One covered open-source projects (presented at MSR 2026); a separate study covered 2,303 context files across Claude Code, Codex, and Copilot. Together, they show varied structures and files that evolve like configuration code. These are configuration mechanisms, shaped by the team and project. The first study also notes vendor recommendations to keep them in version control. (Mohsenimofidi et al.; Chatlatanagulchai et al.)

AGENTS.md, CLAUDE.md and file-based skills serve different purposes, but I recognise a shared design choice: context kept in files that can be versioned and reviewed like code. My YAML repository belongs to that family.

When many practitioners invent the same pattern separately, the pattern is probably load-bearing. A knowledge base in version control isn't documentation. It's closer to instrumentation: a controlled surface for adjusting the model's environment before it acts.

The debate my setup sits inside

My setup sits between two approaches to loading context.

Preloading — decide ahead of time what the model needs, and place it in context. This is what I do. The cost is obvious: I am the retrieval engine. The benefit is just as obvious: what enters the model's attention is chosen, reviewed, and small.

Just-in-time loading — the agent holds lightweight identifiers (file paths, queries) and pulls content into context at runtime. Anthropic described this approach in September 2025. Its account of Claude Code is explicitly hybrid: CLAUDE.md loads up front, while tools such as glob and grep retrieve additional material as needed. (Anthropic)

The evidence behind just-in-time loading isn't only economic. Chroma's context rot research (Hong et al., 2025) reports declining performance as input length grows across its evaluated tasks and models, with the extent depending on the conditions. Having room for information does not guarantee that a model will use it reliably.

Which sharpens the point considerably: selective injection is worth testing for reliability, not only for cost. That is a design inference, rather than proof that a shorter prompt will always be followed better.

But just-in-time loading has a cost the debate sometimes glosses over: it transfers curation to the model, and the model's idea of what's relevant isn't always mine.

My six classes, read as layers

Looking at my own repository through the field's lens, the classes sort into layers I hadn't consciously designed:

So the natural evolution of my system isn't a redesign. It's a loading policy per class: which layers are always injected, and which become pointers the model resolves itself.

The distinction I find most useful: knowledge the model cannot be trusted to seek (conventions, constraints — it won't know to look) versus knowledge it can (snippets, reference material — retrieval is cheap and its selection is adequate). Draw that line per class rather than globally, and the two architectures stop competing and start composing.

The YAML question

I chose YAML because it diffs cleanly, validates mechanically, and lets me build the injection layer with simple tooling. The interesting question is whether the model cares.

This has actually been measured. One preprint used SQL generation as a proxy for agent operations: 9,649 experiments, eleven models and four formats. It found no statistically significant aggregate accuracy difference between YAML, Markdown, JSON and TOON in that setting. That does not establish equivalence across tasks. (McMillan, 2026)

But three second-order effects are real:

  1. Token economy. In McMillan's reported TPC-DS averages, JSON used 28% more tokens than YAML and Markdown used 60% more. These are workload-specific token counts, not universal cost savings. Improving Agents' separate nested-data benchmark found Markdown used fewer tokens than YAML. (Benchmark)
  2. Model sensitivity. He et al. reported variation of up to 40% for GPT-3.5-turbo on a code-translation task, with GPT-4 more robust. Improving Agents found YAML performed best for two of three models in its nested-data stress test. The practical lesson is to test the format with the model and task I actually use. (He et al., 2024)
  3. The subtle one. McMillan observed format affecting which valid interpretation models selected for ambiguous SQL queries. Flat structures and hierarchical ones led models down different join paths in the same data. Structure isn't neutral packaging; it biases resolution.

The study's recommendation is useful here: invest in explicit disambiguation where interpretation matters, not in format optimization.

So YAML is defensible — empirically, not just aesthetically. But the ambiguity finding imposes an obligation: where my YAML encodes domain semantics, the entries need enough prose to disambiguate. The hierarchy alone won't always resolve things the way I intend.

The failure mode I'm watching

The counter-case deserves its own section, because it's the one my setup will eventually meet.

A repository can preserve an obsolete architecture description as faithfully as a current one. Files alone do not decide when a fact expires, which conflicting entry wins, or what should stop reaching the model. For my setup, that makes pruning part of the work: an instruction that once helped can become a source of error.

Git gives me reviewability. It does not give me expiry.

The failure that worries me isn't staleness in general. It's the specific case of contradictions coexisting: two entries disagreeing silently while the model guesses. Which suggests a standing rule, simple and hard to keep:

Every entry is either current or archived. Never both.

Whether that rule survives contact with a growing repository is an open question. I'd rather name it now than discover it in production.

Where this leaves the arrangement

Placing my setup against the field corrects the emphasis rather than the course:

The unresolved question underneath all of it is a division-of-labor question: what should the human curate, and what should the model retrieve?

There is a temptation to answer globally: let the agent retrieve everything. But Anthropic's hybrid example and the task-specific results suggest a more conditional answer. In McMillan's SQL benchmark, file-native retrieval helped the frontier-model group; results for open-source models were mixed.

My suspicion: the durable answer is per-class, not global. And the knowledge most worth curating by hand is precisely the knowledge a model would not think to fetch — constraints, conventions, the reasons behind the reasons.

The rest can be pointed at.

References

End of essayReturn to the archive →