Coaley Peak GEO Research Centre Engine guide
Claude How Anthropic's assistant searches, cites and learns, and how to be its source.
Claude is the AI assistant from Anthropic, widely used for work, writing and coding. It searches the web when that helps, cites its sources in every search answer, and is fed by three documented crawlers that Anthropic says all honour robots.txt. This guide covers its timeline, the four layers behind its answers, its crawlers and how to opt out.
Independent guide: Coaley Peak is not associated with Anthropic. Trademarks
- Owner
- Anthropic
- Launched
- March 2023; claude.ai July 2023; web search March 2025
- Home market
- 185 countries, regions and territories, including the UK and EU
- Languages
- Chats in any language; 11 interface languages
- Open weights
- No
- Crawlers
- ClaudeBot, Claude-SearchBot and Claude-User
Overview
What Claude is, and who uses it
Anthropic was founded by siblings Dario and Daniela Amodei and announced its first funding in May 2021. It published its Constitutional AI method in December 2022 and released Claude in March 2023, first through its API and partner products; the claude.ai chat opened to anyone in the US and UK in July 2023 with Claude 2. The Claude 3 family followed in March 2024, Claude 4 in May 2025, and Claude Opus 5.5 and Sonnet 5.5 in September 2026.
Claude is offered in 185 countries, regions and territories, including the UK and every EU member state, but not China, Hong Kong or Russia. Anthropic does not publish consumer user numbers; its latest business figures, from February 2026, were run-rate revenue of US$14 billion and more than 500 customers spending over US$1 million a year.
Claude gained web search in March 2025 and made it available on every plan worldwide that May. Claude decides when to search, and every search answer includes citations. Anthropic's subprocessor list names Brave Search and TurboPuffer as web search providers, and its help centre says image results are powered by Bing.
For site owners, Claude is among the most clearly documented assistants. Anthropic runs three crawlers, one for training, one for search indexing and one for user requests, says all three honour robots.txt, and has published their IP addresses since April 2026.
Timeline
Claude, from launch to today
Anthropic announces its US$124 million Series A, led by chief executive Dario Amodei and president Daniela Amodei.
Anthropic publishes 'Constitutional AI: Harmlessness from AI feedback', the alignment method later used to train Claude.
Claude is released through Anthropic's API and partner products.
Claude 2 launches with the claude.ai beta, open to anyone in the US and UK.
The Claude 3 family (Haiku, Sonnet and Opus) launches.
Claude 3.5 Sonnet launches.
Claude gains web search with direct citations, for paid users in the US; it reaches every plan worldwide on 27 May 2025.
Anthropic's crawler documentation expands from ClaudeBot alone to three bots, adding Claude-User and Claude-SearchBot.
Claude 4 (Opus 4 and Sonnet 4) launches.
Anthropic agrees a US$1.5 billion settlement with authors over pirated books, about US$3,000 per work; final court approval follows on 20 July 2026.
Anthropic publishes a new constitution for Claude under a CC0 licence, replacing the 2023 version.
Anthropic begins publishing its crawlers' IP addresses at claude.com/crawling/bots.json.
Claude Opus 5.5 launches, followed by Claude Sonnet 5.5 on 28 September.
Things to know
Things to know about Claude
Blocking ClaudeBot only affects future training. Visibility in Claude's search answers depends on Claude-SearchBot and Claude-User, so block those selectively, if at all.
Anthropic says all three of its bots honour robots.txt, including the user-initiated Claude-User, and it supports the non-standard Crawl-delay directive.
Anthropic's older user agents, anthropic-ai and Claude-Web, are deprecated. Current rules should name ClaudeBot, Claude-SearchBot and Claude-User.
Anthropic's subprocessor list names Brave Search and TurboPuffer as web search providers, and its help centre says image results are powered by Bing, so being well indexed by Brave matters for Claude citations.
A noindex robots meta tag keeps a page out of Claude's web search answers, because Anthropic's search partners will not index it or send it to Claude.
Site owners can ask Anthropic to block a specific URL from Claude's web results by emailing webresultsoptout@anthropic.com with proof of ownership.
Claude may use a user's location, inferred from their IP address, for localised search results, which matters for local SEO.
Current Claude models (Opus 5.5, Sonnet 5.5 and Fable 5.1) have a June 2026 knowledge cut-off, so newer pages reach Claude only through live search or fetch.
Claude's API web fetch tool cannot render pages that depend on JavaScript and may return cached content, so key facts belong in server-rendered HTML.
In the API, each citation quotes at most 150 characters of its source, and developers can restrict or exclude domains.
The four layers
How Claude builds an answer
Every AI answer is made in four layers: what the model learned in pre-training, how it was tuned, what it retrieves live, and how it puts the answer together. Here is each one for Claude, and what it means for a brand that wants to be cited.
Layer 1
Pre-training: the base (foundation) model
Claude runs on Anthropic's closed models: Claude Opus 5.5 and Sonnet 5.5 (September 2026), Claude Fable 5.1 and Claude Haiku 4.5. Opus 5.5, Sonnet 5.5 and Fable 5.1 have a published knowledge and training-data cut-off of June 2026. Anthropic describes its training data as a proprietary mix of public online information, public and private datasets, user data and synthetic data. Its public web data comes from ClaudeBot, which it says honours robots.txt and does not access password-protected pages or bypass CAPTCHAs. Consumer chats are used for training only if the user allows it.
What it means for you: With a recent cut-off, Claude may already know a good deal about your brand from what ClaudeBot could crawl. Allowing ClaudeBot, with accurate, substantial pages, is how you shape what future Claude models know.
Layer 2
Post-training: tuning and alignment
Anthropic says Opus 5.5 underwent substantial post-training to make it an effective assistant whose behaviour follows Claude's constitution. Its documented methods include reinforcement learning from human feedback, fine-tuning and Constitutional AI. A new constitution, published in January 2026 under CC0, is used directly in training, and Opus 5.5 is deployed with ASL-3 protections under Anthropic's Responsible Scaling Policy.
What it means for you: Claude's constitution emphasises honesty and balance, so it tends to present trade-offs and cite sources rather than promote. Brands are recommended when credible sources support them, and overstated claims are more likely to be hedged than repeated.
Layer 3
Retrieval: live search
Claude decides when to search and grounds answers in live results. Anthropic's product documentation does not name its index, but its subprocessor list names Brave Search and TurboPuffer for web search, and image results are powered by Bing. Claude-SearchBot indexes pages to improve search quality, and Claude-User fetches pages users ask about. Claude may localise results using the user's approximate location.
What it means for you: Being well indexed by Brave Search, as well as reachable by Claude-SearchBot and Claude-User, is the practical route into Claude's answers. A page that Brave cannot see, or that carries noindex, will not be cited.
Layer 4
The answer: what people read
Anthropic says every search response in Claude includes citations: direct citations to sources, source links for further reading and relevant quotes where appropriate, and each image result links to its source. In the API, citations are always on for web search, each quoting up to 150 characters of the source, and developers who show results to end users must display them.
What it means for you: Claude cites short, exact passages. Pages whose key claims fit in a sentence or two, with your brand named in them, give Claude something precise to quote and link.
Crawlers
Claude's crawlers and user agents
The user-agent tokens to look for in your server logs and to name in robots.txt.
| User-agent token | What it does | robots.txt |
|---|---|---|
| ClaudeBot Documentation | Collects web content that could potentially contribute to training Anthropic's models. Restricting it signals that a site's future materials should be excluded from training datasets. | Follows robots.txt, including Crawl-delay. Rules are needed on every subdomain. |
| Claude-SearchBot Documentation | Navigates the web to improve the quality of Claude's search results. Disabling it may reduce a site's visibility and accuracy in Claude's search answers. | Follows robots.txt, Anthropic says. |
| Claude-User Documentation | Fetches pages when a user asks Claude a question. Disabling it stops Claude retrieving your content for users, which may reduce your visibility in user-directed search. | Follows robots.txt, Anthropic says, even though it is user-initiated. |
Anthropic does not publish full user-agent strings. Commonly observed strings follow the pattern 'compatible; ClaudeBot/1.0; +claudebot@anthropic.com', so match on the token. The older anthropic-ai and Claude-Web tokens are deprecated.
Crawl sources
Where Claude crawls from, and how to verify it
Since April 2026 Anthropic has published its crawlers' IP addresses at claude.com/crawling/bots.json; if a crawler's source IP is on the list, Anthropic says, it is coming from Anthropic. When we checked on 6 October 2026 the file listed 28 IPv4 prefixes, such as 216.73.216.0/22, and no IPv6. Until March 2026 Anthropic said it did not publish IP ranges because it used service providers' public IPs.
Anthropic publishes no reverse DNS scheme or network. Verify by matching the user-agent token and checking the IP against the current list, which changes over time.
Anthropic warns that blocking its IP addresses may not reliably opt you out, because it stops Anthropic reading your robots.txt. Suspected crawler problems should be reported to claudebot@anthropic.com from an email address at the domain concerned.
Opting out
How to opt out of Claude, or limit what it uses
Refuse training, stay citable
Disallow ClaudeBot to keep future content out of training, and leave Claude-SearchBot and Claude-User allowed. Add the rules on every subdomain. Blocking ClaudeBot is not retroactive.
Official pageSlow crawling down
Anthropic supports Crawl-delay, for example User-agent: ClaudeBot with Crawl-delay: 1.
Official pageKeep a page out of search answers
A noindex robots meta tag tells Anthropic's search partners not to index the page, so it will not appear in Claude's web search answers.
Official pageAsk for a URL to be blocked
Email webresultsoptout@anthropic.com with proof that you own the URL, such as WHOIS details, a domain payment receipt or an email from an address at the domain.
Official pageReport unlawful content
Use Anthropic's content reporting form; copyright and trademark disputes have a separate process.
Official pagePrivacy requests about training data or answers
Email privacy@anthropic.com with your name, screenshots, the prompt and the URLs concerned. Anthropic says rights over training data are limited and that correcting model outputs may not always be possible.
Official pageKeep your own chats out of training
Free, Pro and Max users can switch off the use of their chats for training in Privacy Settings.
Official page
Opting out of ClaudeBot works going forward only: Claude models already trained keep what they learned. Blocking Claude-SearchBot or Claude-User, or adding noindex, costs you visibility in Claude's search answers, so most brands should leave them allowed.
How we help
How we get brands cited in Claude
The Coaley Peak GEO Research Centre researches and implements GEO for SEO agencies and in-house teams, with foreign markets as our speciality.
Claude visibility audit
We ask Claude the questions your customers ask, in each market and language, and record whether you are cited, what it quotes and how it describes you.
Crawler policy for all three bots
We set robots.txt so Claude-SearchBot and Claude-User can reach you, with ClaudeBot allowed or refused as you choose, and verify the result against Anthropic's published IPs.
Brave Search visibility
Brave Search is among Claude's named search providers. We check and improve how Brave indexes and ranks you, an engine most SEO teams never look at.
Quotable passages
Claude quotes short, exact passages. We rewrite key pages so each claim is stated plainly, with your brand named, in server-rendered HTML.
Measurement
We track your citations in Claude alongside ChatGPT, Gemini and Perplexity, so you can see where each assistant sends people.
More on our research, services and team at the Coaley Peak GEO Research Centre.
FAQ
Claude GEO: questions
What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?
ClaudeBot collects web content that may be used for training. Claude-SearchBot indexes pages to improve Claude's search results. Claude-User fetches pages when a user asks. Each has its own robots.txt token, and Anthropic says all three honour robots.txt.
How do I stop Claude training on my site without losing visibility?
Disallow ClaudeBot and leave Claude-SearchBot and Claude-User allowed. Blocking ClaudeBot signals that future content should be excluded from training, without affecting Claude's search answers.
Where does Claude get its search results?
Anthropic's product docs do not name its index, but its subprocessor list names Brave Search and TurboPuffer as web search providers, and its help centre says image results are powered by Bing. Claude-SearchBot also indexes pages for search quality.
Does Claude cite its sources?
Yes. Anthropic says every search response includes citations, with source links and relevant quotes. In the API, citations are always on for web search, each quoting up to 150 characters of its source.
What is Claude's knowledge cut-off?
Anthropic publishes June 2026 as the knowledge and training-data cut-off for Claude Opus 5.5, Sonnet 5.5 and Fable 5.1. Claude Haiku 4.5's reliable knowledge runs to February 2025. Newer information reaches Claude through search.
Where do Anthropic's crawlers come from?
Anthropic has published its crawlers' IP addresses at claude.com/crawling/bots.json since April 2026. It publishes no reverse DNS scheme, and warns that blocking its IPs can stop it reading your robots.txt.
Want a free chat about Claude?
Tell us which market and brand you are working on. If we can help, we will set up a free, no-obligation chat with Stephen or Ethan.
More engine guides
Related engines
Independence and trademarks. Claude and Anthropic are trademarks of Anthropic, PBC. Coaley Peak is independent: we are not associated with, endorsed by or a partner of Anthropic. This guide is our own research from public documentation and reputable reporting, checked in October 2026.
Reviewed by Stephen Grindley, research centre lead. Last checked .
Document reference: ISO_webpage_geo_engine_claude_v1
Last modified: 7 October 2026
GEO Research Centre·Engine guides·Claude