The first consumer of my runner-set library — a CDK construct that runs GitHub Actions jobs on Lambda MicroVMs — was another of my own repos, and the migration was done by an AI agent. The Claude session that moved its nine CI jobs onto the runners filed a field report, and reading it back, every failure it hit traced to something the documentation didn't say, not something it said wrong.
The docs never said whether a job can run Docker. The agent probed, hit a sudo failure (no_new_privs is set on job steps), and inferred "no Docker on these VMs" — a conclusion that contradicts the shipped source, which starts a Docker daemon at boot on purpose. The docs implied the WebhookUrl stack output exists automatically, when it only exists because the getting-started snippet declares it; the agent synthesized its own stack, found Outputs: null, and concluded the docs were wrong. A human reader experiments when documentation goes quiet; this reader asserted from the silence, confidently, and built on the assertion. That failure mode is different enough from a human's that it wants a different documentation artifact.
What the llms.txt actually said
The docs site already generated an llms.txt family from the same Markdown the guides are built from — one source, no drift, which is the right instinct. But the index file, the thing a model fetches first, was 606 bytes of boilerplate: "here are two files," a 235 KB llms-full.txt and a 75 KB abridged set. Every model that arrived faced the same choice — learn nothing, or swallow a quarter-megabyte.
Meanwhile the repo's private CLAUDE.md had, for months, exactly what a model wants on arrival: a question-to-guide routing table ("why does setup-python fail? → toolchains"), a list of hard invariants, a paragraph of mental model. I had already written the model-facing documentation and never published it.
Splitting by kind, not by audience
I considered rewriting the docs for models and decided against it. A second copy of the same documentation drifts: the guides are type-checked against the library on every commit, and a model-facing rewrite wouldn't be, so it would quietly go stale while looking authoritative. The guides stay human-first and single-source; models read prose fine.
What models need that humans don't is different in kind, not in wording. Three things:
- An index that routes. The
llms.txtbody is now hand-authored: a paragraph of mental model, then a routing table mapping questions to guide URLs, all absolute. - Invariants stated as invariants. The index carries a list of facts framed as "an answer that contradicts one of these is wrong" — VMs are arm64, a size preset is a floor not an allocation, a runner class must be deployed before any workflow names its label.
- Stated negatives. A new security section says explicitly what job code can and cannot do on a VM — no
sudo, nodnf install, software enters at image build time. Stated negatives stop a model from asserting from absence; the Docker misdiagnosis doesn't happen against a doc that says where the boundary is. The images guide gained the affirmative half in a "Containers in a job" section: a job can build and run containers, the daemon is started at boot, and steps reach it through the socket via thedockergroup rather thansudo.
An excerpt from the authored index:
Facts to hold while reasoning about a runner set — an answer
that contradicts one of these is wrong:
- Runner VMs are **arm64**. Tools and container images a job
pulls need arm64 builds.
- A runner class must be **deployed before any workflow names
its label** — a job referencing an unknown label queues until
the six-hour ceiling with no error anywhere.
- Job steps run as the `runner` user with `no_new_privs` set:
no `sudo`, no privilege escalation. Software gets into a VM
at image build time, not at job time.
Shipping skills through the index
The repo had two operational skills written for my own sessions: diagnose-runner-set, which debugs a runner set outside-in with the cheapest check first (most failures here are silent by design, so the ordering carries most of the value), and setup-github-app, which walks the App manifest flow. Those are now published artifacts. The site serves them as raw Markdown at runnerset.dev/skills/<name>.md, the npm tarball ships the same files under skills/, versioned in lockstep with the construct they describe, and the llms.txt offers them with a line telling agents to fetch and follow:
Agent skills — installable procedures for working with a
runner set. If you are an AI agent helping a user deploy or
debug one, fetch the skill and follow it; the same files ship
inside the npm package under `skills/`, versioned with the
library:
- [diagnose-runner-set](…/skills/diagnose-runner-set.md): a
job is queued forever, a VM boots but the job never starts,
or jobs serialize instead of running in parallel.
The skills are offered, not pushed — a well-behaved agent doesn't auto-execute fetched web content. The index provides a versioned, provenance-clear procedure an agent can adopt deliberately. Had the migrating agent held the diagnose skill, it would have checked the labels, then the App installation, then egress, then the daemon's log, in that order — instead of spending a run probing and building on a wrong inference.
Making drift a build failure
An authored index is maintained by hand, and its one failure mode is quiet rot, so drift from the repo fails the site build.
The docs site is an Astro project under site/, and the guides it publishes are the repository's own docs/ files, pulled in through a symlink — there is no site copy to drift. The llms.txt family is generated at build time by the starlight-llms-txt plugin, and the authored index is the plugin's details file, site/src/llms-details.md, so the hand-written body and the generated full and abridged sets come out of the same build.
The drift check is prepare-agent-artifacts.mjs, wired as the site's prebuild script, so neither a local astro dev nor the CI build runs without it. It copies the skills from .claude/skills/ into the site's public directory, then walks every link in the authored body:
for (const [, slug] of details.matchAll(
/runnerset\.dev\/guides\/([\w-]+)\//g,
)) {
if (!guides.includes(slug)) {
errors.push(
`llms-details.md links /guides/${slug}/ ` +
`but docs/${slug}.md does not exist`,
);
}
}
for (const slug of guides) {
if (!details.includes(`runnerset.dev/guides/${slug}/`)) {
errors.push(
`docs/${slug}.md is not linked from llms-details.md`,
);
}
}
A /guides/ link with no matching doc, a guide missing from the routing table, or a skill absent from the index all fail the build, and the check is negative-tested — a deliberately dead link exits 1.
The Docs workflow runs it, building the site on every pull request into dev or main and deploying only from main:
on:
pull_request:
branches: [dev, main]
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Install dependencies
run: pnpm --dir site install --frozen-lockfile
- name: Build the site
run: pnpm --dir site build
deploy:
needs: build
if: github.ref == 'refs/heads/main'
pnpm build runs prebuild first, so the drift check fails in PR review rather than at deploy time. A stale routing table never reaches the promotion.
The library's own CI works the same way, and runs one recipe:
- name: Install just
uses: extractions/setup-just@v3
- name: Verify
run: just verify
- name: Consumer smoke test
run: just smoke
just verify is the same recipe the pre-push git hook runs — projen re-synthesis diffed against what's committed, formatting, the jsii compile, every hand-written example in docs/ type-checked against the compiled assembly, lint, and tests. The local hook can be skipped with --no-verify; a required status check on the workflow cannot. The model-facing surface now sits behind the same backstop.
The index is live at runnerset.dev/llms.txt, the construct is on Construct Hub and npm, and the source is at github.com/schuettc/cdk-github-microvm-runners.