robots.txt #36
inhji
commented
Owner
- Generate /sitemap.xml listing canonical URLs, keep it updated on publish, and reference it from /robots.txt.
- Add explicit User-agent entries for AI crawlers (GPTBot, OAI-SearchBot, Claude-Web, Google-Extended) with allow/disallow rules that match your policy.
- Add Content-Signal directives to your robots.txt declaring preferences for ai-train, search, and ai-input. For example: Content-Signal: ai-train=no, search=yes, ai-input=no
- Generate /sitemap.xml listing canonical URLs, keep it updated on publish, and reference it from /robots.txt.
- Add explicit User-agent entries for AI crawlers (GPTBot, OAI-SearchBot, Claude-Web, Google-Extended) with allow/disallow rules that match your policy.
- Add Content-Signal directives to your robots.txt declaring preferences for ai-train, search, and ai-input. For example: Content-Signal: ai-train=no, search=yes, ai-input=no
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?