Skip to content

Search engines and AI crawlers

What the server tells search engines and AI crawlers, the settings that change it, and how to verify what it serves.

robots.txt and meta tags are statements that crawlers may or may not honor; they are not a technical block. If you need to block, use a WAF or rate limiting.

Scope Setting Where to change it When not set
Server search_indexing (allow / deny) The “Search engines” and “AI” sections on the “Search and crawlers” page of the admin server settings, my instance set, PATCH /api/v1/admin/settings (Instance settings) deny
Server ai_usage (allow / deny_training / deny) Same deny
Server indexable_default (the indexable of new accounts) Same false
Server security_contact (where to report vulnerabilities) Same Empty (no security.txt is served)
Person indexable (appear in search) “Search engines and AI” in the profile settings, PATCH /api/v1/accounts/update_credentials indexable_default at registration
Person ai_opt_out (refuse AI training and use) Same false
  • A page’s statement is the stricter of the server’s setting and the page owner’s setting. A person cannot allow more than the server does.
  • Search engines are told they may index only the profiles and public posts of accounts with indexable = true, on a server with search_indexing = allow. A server that wants to be searchable changes both search_indexing and indexable_default. People who registered before you changed indexable_default are not included until they choose indexable themselves.
  • A changed setting takes effect in /robots.txt, pages and media responses without a restart.
Response When refusing search When refusing AI
Pages of profiles and public posts noindex, noarchive in <meta name="robots"> and X-Robots-Tag noai, noimageai, Content-Usage: train-ai=n (train-ai=n, ai-use=n for deny), and tdm-reservation: 1 (header and meta)
unlisted posts, 404 pages for hidden or nonexistent pages, pages of remote accounts, app screens (/home, /settings and so on) noindex, noarchive whatever the setting Follows the server setting
/, /about, /about/crawler Follows the server setting Follows the server setting
/media/ (when this server serves it) Follows the server setting. No per-owner statement Follows the server setting. No per-owner statement
The client API, ActivityPub documents, /.well-known/, /nodeinfo/, sign-in and OAuth screens X-Robots-Tag: noindex Not added (robots.txt is enough)

noindex does not stop link previews (see Link previews).

GET /robots.txt is built from the server settings and the bundled list of AI crawlers (text/plain; charset=utf-8, Cache-Control: public, max-age=3600, ETag). Operators cannot add rules of their own.

  • If ai_usage is deny_training or deny, training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and so on) get Disallow: /.
  • If ai_usage is deny_training or deny, a User-agent: __usage_ai-training group with Disallow: / states a preference against AI training for every path, whichever crawler fetches the content (the draft-kelley-aipref-training-preference-signal Internet-Draft). It names a use, not a crawler, so it stops no fetches. With ai_usage = allow the group is left out, which states no preference, because its Allow: / would state a preference in favor of training for every path, including the pages of people who set ai_opt_out.
  • If ai_usage is deny, the crawlers of AI search, assistants and agents (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot and so on) get Disallow: / too.
  • All crawlers (User-agent: *) are refused only /api/, /oauth/, /auth/, /ap/ and /media/proxy/, and the settings are stated in Content-Usage (the IETF AI Preferences draft) and Content-Signal (Content Signals) lines. Refusing search does not produce Disallow: /: blocking fetches would stop crawlers from reading noindex, and would also stop the link-preview fetchers that obey robots.txt.
  • /.well-known/ and /nodeinfo are not refused, so that server lists and statistics keep working.
  • When account_domain is separate from the host, the host side serves it. The robots.txt of the handle’s domain is the operator’s own business.
Path Served when Content
/.well-known/tdmrep.json ai_usage is anything but allow [{"location": "/*", "tdm-reservation": 1}] (TDM Reservation Protocol). 404 for allow
/.well-known/security.txt security_contact is set Contact, Expires (30 days after the time of the response), Preferred-Languages (default_locale and en), Canonical (RFC 9116)

The instance information page (/about) shows whether search and AI are allowed, and what the statement means, from search_indexing and ai_usage in GET /api/v1/instance. The explanation places the statements against Article 30-4 of the Japanese Copyright Act (such statements may not be treated as a legal refusal) and Article 4 of the EU DSM Directive (a machine-readable reservation of text and data mining). It does not guarantee any legal effect.

These settings are about what crawls this server. The server also crawls others for the cards of links in posts, as my-linkcard: it obeys robots.txt and the directives of the linked pages and limits its requests per host, as link previews describes. That crawler is set with link_cards, link_cards_ignore_robots_txt, and, per domain, with link card domains.

The operators of those sites read about it at https://<host>/about/crawler, which /about links to (the page for site operators). With link_cards_ignore_robots_txt on, they cannot opt out with robots.txt, and the page says so: they can only limit the card with the directives of their pages, or ask you for a rule on their domain.

  • A person’s ai_opt_out is stated only on pages this server serves. Media has no owner in its key, and some setups let a CDN serve it directly, so media carries only the server-wide statement. Statements on pages served by other servers are left to those servers (no custom ActivityPub extension is sent).
  • ai_usage = deny also states a refusal of the actions in which a user of ChatGPT or a similar service has it read a page of this server.
  • The indexable of a remote account (Mastodon’s toot:indexable) is received and stored. Pages of remote accounts are always noindex on this server.
  • indexable also decides whether a person’s posts are in this server’s own full-text search, on a server with a search engine. There it covers the visibilities of search_visibilities, which can include local. search_indexing has no effect on that search, and what search engines are told stays as this page says.

Serving media from object storage or a CDN

Section titled “Serving media from object storage or a CDN”

When object storage or a CDN serves media directly, /media/ responses do not pass through this server and do not carry the headers in the table above. The server-wide statement still applies to everything through /robots.txt; to add the headers too, add the following to /media/* in the CDN’s response-header rules (an example for ai_usage = deny).

X-Robots-Tag: noindex, noarchive, noai, noimageai
Content-Usage: train-ai=n, ai-use=n
tdm-reservation: 1

When you change the settings, change the CDN rules to match. /media/proxy/ is served by this server, so it needs no rule.

The list is generated from robots.json of ai-robots-txt/ai.robots.txt (MIT) and bundled in the binary. It is not fetched at run time, so a newer list comes with an upgrade.

Terminal window
curl -sS https://<host>/robots.txt
curl -sS -o /dev/null -D - -H 'Accept: text/html' https://<host>/@alice | grep -iE 'x-robots-tag|content-usage|tdm-reservation'
curl -sS https://<host>/.well-known/tdmrep.json
curl -sS https://<host>/.well-known/security.txt
  • On a server with no settings, /robots.txt has User-agent: GPTBot, User-agent: __usage_ai-training and Content-Usage: search=n, train-ai=n, ai-use=n, and profiles carry X-Robots-Tag: noindex, noarchive, noai, noimageai.
  • With search_indexing = allow and ai_usage = allow, the profile of an account that chose indexable carries no X-Robots-Tag, and /.well-known/tdmrep.json is 404.

For LLMs and coding agents:this page as Markdown, llms.txt,llms-full.txt, llms-small.txt

Analytics on this site