Search engines and AI crawlers
What the server tells search engines and AI crawlers, the settings that change it, and how to verify what it serves.
robots.txt and meta tags are statements that crawlers may or may not honor; they are not a technical block. If you need to block, use a WAF or rate limiting.
Settings
Section titled “Settings”| Scope | Setting | Where to change it | When not set |
|---|---|---|---|
| Server | search_indexing (allow / deny) |
The “Search engines” and “AI” sections on the “Search and crawlers” page of the admin server settings, my instance set, PATCH /api/v1/admin/settings (Instance settings) |
deny |
| Server | ai_usage (allow / deny_training / deny) |
Same | deny |
| Server | indexable_default (the indexable of new accounts) |
Same | false |
| Server | security_contact (where to report vulnerabilities) |
Same | Empty (no security.txt is served) |
| Person | indexable (appear in search) |
“Search engines and AI” in the profile settings, PATCH /api/v1/accounts/update_credentials |
indexable_default at registration |
| Person | ai_opt_out (refuse AI training and use) |
Same | false |
- A page’s statement is the stricter of the server’s setting and the page owner’s setting. A person cannot allow more than the server does.
- Search engines are told they may index only the profiles and
publicposts of accounts withindexable = true, on a server withsearch_indexing = allow. A server that wants to be searchable changes bothsearch_indexingandindexable_default. People who registered before you changedindexable_defaultare not included until they chooseindexablethemselves. - A changed setting takes effect in
/robots.txt, pages and media responses without a restart.
What the server returns
Section titled “What the server returns”| Response | When refusing search | When refusing AI |
|---|---|---|
Pages of profiles and public posts |
noindex, noarchive in <meta name="robots"> and X-Robots-Tag |
noai, noimageai, Content-Usage: train-ai=n (train-ai=n, ai-use=n for deny), and tdm-reservation: 1 (header and meta) |
unlisted posts, 404 pages for hidden or nonexistent pages, pages of remote accounts, app screens (/home, /settings and so on) |
noindex, noarchive whatever the setting |
Follows the server setting |
/, /about, /about/crawler |
Follows the server setting | Follows the server setting |
/media/ (when this server serves it) |
Follows the server setting. No per-owner statement | Follows the server setting. No per-owner statement |
The client API, ActivityPub documents, /.well-known/, /nodeinfo/, sign-in and OAuth screens |
X-Robots-Tag: noindex |
Not added (robots.txt is enough) |
noindex does not stop link previews (see Link previews).
/robots.txt
Section titled “/robots.txt”GET /robots.txt is built from the server settings and the bundled list of AI crawlers (text/plain; charset=utf-8, Cache-Control: public, max-age=3600, ETag).
Operators cannot add rules of their own.
- If
ai_usageisdeny_trainingordeny, training crawlers (GPTBot,ClaudeBot,CCBot,Google-Extended,Applebot-Extendedand so on) getDisallow: /. - If
ai_usageisdeny_trainingordeny, aUser-agent: __usage_ai-traininggroup withDisallow: /states a preference against AI training for every path, whichever crawler fetches the content (thedraft-kelley-aipref-training-preference-signalInternet-Draft). It names a use, not a crawler, so it stops no fetches. Withai_usage = allowthe group is left out, which states no preference, because itsAllow: /would state a preference in favor of training for every path, including the pages of people who setai_opt_out. - If
ai_usageisdeny, the crawlers of AI search, assistants and agents (OAI-SearchBot,ChatGPT-User,Claude-User,PerplexityBotand so on) getDisallow: /too. - All crawlers (
User-agent: *) are refused only/api/,/oauth/,/auth/,/ap/and/media/proxy/, and the settings are stated inContent-Usage(the IETF AI Preferences draft) andContent-Signal(Content Signals) lines. Refusing search does not produceDisallow: /: blocking fetches would stop crawlers from readingnoindex, and would also stop the link-preview fetchers that obey robots.txt. /.well-known/and/nodeinfoare not refused, so that server lists and statistics keep working.- When
account_domainis separate from the host, the host side serves it. The robots.txt of the handle’s domain is the operator’s own business.
/.well-known/
Section titled “/.well-known/”| Path | Served when | Content |
|---|---|---|
/.well-known/tdmrep.json |
ai_usage is anything but allow |
[{"location": "/*", "tdm-reservation": 1}] (TDM Reservation Protocol). 404 for allow |
/.well-known/security.txt |
security_contact is set |
Contact, Expires (30 days after the time of the response), Preferred-Languages (default_locale and en), Canonical (RFC 9116) |
Information page
Section titled “Information page”The instance information page (/about) shows whether search and AI are allowed, and what the statement means, from search_indexing and ai_usage in GET /api/v1/instance.
The explanation places the statements against Article 30-4 of the Japanese Copyright Act (such statements may not be treated as a legal refusal) and Article 4 of the EU DSM Directive (a machine-readable reservation of text and data mining).
It does not guarantee any legal effect.
This server’s own fetching
Section titled “This server’s own fetching”These settings are about what crawls this server. The server also crawls others for the cards of links in posts, as my-linkcard: it obeys robots.txt and the directives of the linked pages and limits its requests per host, as link previews describes.
That crawler is set with link_cards, link_cards_ignore_robots_txt, and, per domain, with link card domains.
The operators of those sites read about it at https://<host>/about/crawler, which /about links to (the page for site operators). With link_cards_ignore_robots_txt on, they cannot opt out with robots.txt, and the page says so: they can only limit the card with the directives of their pages, or ask you for a rule on their domain.
Limits
Section titled “Limits”- A person’s
ai_opt_outis stated only on pages this server serves. Media has no owner in its key, and some setups let a CDN serve it directly, so media carries only the server-wide statement. Statements on pages served by other servers are left to those servers (no custom ActivityPub extension is sent). ai_usage = denyalso states a refusal of the actions in which a user of ChatGPT or a similar service has it read a page of this server.- The
indexableof a remote account (Mastodon’stoot:indexable) is received and stored. Pages of remote accounts are alwaysnoindexon this server. indexablealso decides whether a person’s posts are in this server’s own full-text search, on a server with a search engine. There it covers the visibilities ofsearch_visibilities, which can includelocal.search_indexinghas no effect on that search, and what search engines are told stays as this page says.
Serving media from object storage or a CDN
Section titled “Serving media from object storage or a CDN”When object storage or a CDN serves media directly, /media/ responses do not pass through this server and do not carry the headers in the table above.
The server-wide statement still applies to everything through /robots.txt; to add the headers too, add the following to /media/* in the CDN’s response-header rules (an example for ai_usage = deny).
X-Robots-Tag: noindex, noarchive, noai, noimageaiContent-Usage: train-ai=n, ai-use=ntdm-reservation: 1When you change the settings, change the CDN rules to match. /media/proxy/ is served by this server, so it needs no rule.
The AI crawler list
Section titled “The AI crawler list”The list is generated from robots.json of ai-robots-txt/ai.robots.txt (MIT) and bundled in the binary. It is not fetched at run time, so a newer list comes with an upgrade.
Verify
Section titled “Verify”curl -sS https://<host>/robots.txtcurl -sS -o /dev/null -D - -H 'Accept: text/html' https://<host>/@alice | grep -iE 'x-robots-tag|content-usage|tdm-reservation'curl -sS https://<host>/.well-known/tdmrep.jsoncurl -sS https://<host>/.well-known/security.txt- On a server with no settings,
/robots.txthasUser-agent: GPTBot,User-agent: __usage_ai-trainingandContent-Usage: search=n, train-ai=n, ai-use=n, and profiles carryX-Robots-Tag: noindex, noarchive, noai, noimageai. - With
search_indexing = allowandai_usage = allow, the profile of an account that choseindexablecarries noX-Robots-Tag, and/.well-known/tdmrep.jsonis 404.
For LLMs and coding agents:this page as Markdown, llms.txt,llms-full.txt, llms-small.txt