I fell down this rabbit hole while reading why publishers keep complaining about AI crawlers. One vendor's data: 1.9 billion crawls ignored robots.txt rules in a six-month window. Another measured a 70,900:1 crawl-to-referral ratio for one major AI provider. AI bots averaged 4.2% of all HTML requests in 2025, peaking at 6.4%.
The rules are right there in a text file. The file isn't the problem. It's that a text file can't prove who wrote it, can't be revoked, and isn't bound to the domain it claims to speak for.
So I built AIFeed: an open, signed way for a site to declare what AI agents may do with its content. Here's what it does, what I measured, and what's still missing.
No proof. Anyone can copy a robots.txt to another domain, or edit the one on a compromised host. An agent has no way to check whether the rules it just read actually came from the owner.
No revocation. Once a crawler caches your "allowed for training" file, changing it later does nothing. There is no revocation list and no status to re-check on reuse.
No binding. robots.txt has no notion of "this domain, this key, this revision". Nothing chains the domain to the permissions.
I wanted something an agent could verify on its own, without calling a service and without trusting a new intermediary.
A publisher generates an Ed25519 key pair and publishes a manifest at /.well-known/ai.json. The manifest lists per-use permissions (training, retrieval, quoting, summarization), crawl rules, licensing, and a revision number. It's signed over JCS-canonical JSON, and the public key is anchored in a DNS TXT record (_aifeed).
An agent verifying a site walks that chain: TLS → domain → signature → DNS anchor. Then it re-checks a multi-signature revocation registry on every use. If the key was rotated or revoked, verification says so.
For delivery there are two profiles. AIFeed Markdown is the native one (.aifeed.md, text/aifeed+markdown) with a permission block in-band. MAKO is supported as a compatibility profile, so the same tooling can serve both. There's a signed delta index too, so an agent that already has unchanged pages doesn't refetch them.
All of it is zero-dependency. The reference implementation is Node standard library; the independent Python verifier is stdlib-only. No accounts, no SaaS in the middle.
From the local benchmark harness (60-page corpus, 100 tenants, 4 client profiles): −68.83% transferred bytes serving markdown profiles instead of HTML −95.73% when a client uses the delta index and only 10% of pages changed −55.19% origin bytes and −56.23% origin CPU on the publisher side 0.70 ms to verify one page's signature
Caveat, and I'll keep repeating it: this is a simulation harness, not a field trial. The 30-day production pilot hasn't run. The vendor claims of 90%+ token savings you may have seen require semantic summarization, which AIFeed deliberately does not do automatically.
There's also an MCP server (npx -y aifeed-mcp-server) with tools for verifying manifests, fetching token-budgeted markdown, listing assets and verifying their digests. It's listed in the official MCP registry and on Smithery.
Eight live demo origins are up at https://aifeed.md if you want to poke at real signed manifests: a news site, a docs site, a store, and more.
What's still missing No external cryptographic review yet, and the specs are a draft, not frozen. Origin + DNS compromise is undetectable on first contact. A signature proves the key signed the file, not that the key was safe. Legal review of the permission semantics is pending; the paper labels those claims as needing review. The 30-day pilot hasn't happened. If you run a site with real crawler traffic, I'd like to talk.
Where to look Specs, tooling, conformance vectors: https://github.com/denyn1/aifeed-protocol Paper draft: https://aifeed.md/aifeed-preprint.pdf npm: @aifeed/verify, aifeed, aifeed-mcp-server, @aifeed/frameworks PyPI: aifeed
If you try it and something breaks, open an issue. There are a few good first issues labeled if you want to help. And if you verify your own domain and get "no manifest found", that's not a bug. Most of the web is still unsigned.