Coaley Peak GEO Research Centre Engine guide
Open-weight models The models inside everyone else's products, and how they learn about a brand.
Open-weight models are language models whose weights anyone can download and run. Llama, Gemma, Qwen, DeepSeek, Mistral and Falcon sit inside thousands of apps, agents and company tools, answering from what they learned in training. This guide covers what they learned from, their cut-offs and licences, the crawlers that feed them, and how to influence them.
Independent guide: Coaley Peak is not associated with Meta, Google, Alibaba, DeepSeek, Mistral AI, TII and the Common Crawl Foundation. Trademarks
- Owner
- Meta, Google, Alibaba, DeepSeek, Mistral AI, TII and the Common Crawl Foundation
- Launched
- LLaMA February 2023; Common Crawl archiving since 2008
- Home market
- Worldwide, inside third-party apps, agents and company tools
- Languages
- Many: Qwen3 119, Gemma 140+, Llama 4 200 in pre-training
- Open weights
- Yes, by definition, under licences from Apache 2.0 to custom
- Crawlers
- CCBot for Common Crawl, plus meta-externalagent, Google-Extended and MistralAI-Training
Overview
What Open-weight models is, and who uses it
Open-weight models answer from memory. Unlike ChatGPT or Perplexity, a downloaded model has no search engine attached unless the company running it adds one. Whatever it says about your brand comes from its training data, frozen at a cut-off date, and every copy carries the same knowledge, right or wrong, wherever it is deployed.
Their reach is large and hard to see. Meta says Llama had been downloaded more than a billion times by March 2025, and Google says Gemma has passed 400 million downloads, with more than 100,000 community variants. They run in customer-service bots, internal company assistants, coding tools and phone apps, most of which never say which model they use.
Much of what they know comes from one place: Common Crawl, a non-profit that has archived the web since 2008 and publishes a new crawl roughly every month. Datasets filtered from it, such as C4, RefinedWeb, FineWeb, Dolma, DCLM and Nemotron-CC, supply much of the web text these models learn from. A Mozilla Foundation study found that at least 64% of 47 large language models published between 2019 and October 2023 used filtered Common Crawl data.
That is why base model targeting, our specialist service, starts here. If a brand is well represented in the pages that survive into these datasets, open-weight models are more likely to know it correctly. If it is not, no amount of conventional SEO will change what a model already deployed believes.
Timeline
Open-weight models, from launch to today
Gil Elbaz founds Common Crawl as a non-profit open archive of web crawl data. Crawling begins in 2008.
Google's T5 paper introduces C4, a cleaned training corpus built from a Common Crawl snapshot.
Meta's LLaMA paper shows a model trained only on public data, 67% of it English Common Crawl and 15% C4.
TII's Falcon 40B, trained mainly on RefinedWeb, a filtered Common Crawl dataset, is available under Apache 2.0.
Meta releases Llama 2 with a licence that allows commercial use.
Mistral AI releases Mistral 7B under Apache 2.0.
Google launches Gemma, its family of open-weight models.
Hugging Face publishes FineWeb, a 15-trillion-token dataset filtered from 96 Common Crawl snapshots.
DeepSeek releases DeepSeek-R1 under the MIT licence, explicitly allowing commercial use and distillation.
Meta releases Llama 4 Scout and Maverick, with a pre-training data cut-off of August 2024.
Alibaba releases Qwen3 under Apache 2.0, trained on about 36 trillion tokens in 119 languages and dialects.
Common Crawl publishes an opt-out registry of legal removal requests.
Google releases Gemma 4, the first Gemma under Apache 2.0, with a pre-training cut-off of January 2025.
DeepSeek releases DeepSeek-V4 under the MIT licence, pre-trained on more than 32 trillion tokens.
Alibaba releases Qwen3.8, including a 27-billion-parameter model under Apache 2.0.
Common Crawl publishes CC-MAIN-2026-39: 2.17 billion pages from 40.4 million hosts, crawled between 4 and 17 September 2026.
Things to know
Things to know about Open-weight models
Brand facts newer than a model's cut-off cannot be in its weights. Published cut-offs: Llama 4, August 2024; Llama 3.1, December 2023; Gemma 4, January 2025; Gemma 3, August 2024. Qwen and Falcon publish none, and DeepSeek dates its data only in its EU training summaries.
CCBot does not run JavaScript or use cookies. Content that only appears after client-side rendering never reaches Common Crawl or the datasets built from it.
Common Crawl prioritises URLs by Harmonic Centrality, a measure of how close a host sits to the core of the web's link graph. Well-linked hosts are crawled more often.
Blocking CCBot only works going forward. Common Crawl's archives are immutable, so pages already captured stay in past crawls and in the datasets built from them.
Quality filters decide which crawled pages reach a model. The original LLaMA kept Common Crawl pages that a classifier judged similar to pages cited as Wikipedia references; FineWeb-Edu kept pages a classifier scored as educational.
Licences fall into three groups: Apache 2.0 (Gemma 4, Qwen3, Mistral Large 3, Falcon 40B), MIT (DeepSeek-R1 and V4) and custom (Llama 4, Gemma 1 to 3, Mistral Medium 3.5, TII's Falcon licence). Most custom licences still allow commercial use.
Llama 4's licence does not grant rights to its multimodal models to individuals or companies based in the EU, though end users of products built on them are unaffected.
Under the EU AI Act, providers of general-purpose models must publish a training-content summary listing the most relevant domains they scraped: the top 10% by content size, or for smaller companies the top 5% or 1,000 domains, whichever is lower.
Spoofed CCBots are common. Verify requests against Common Crawl's published IP list and with forward-confirmed reverse DNS.
Common Crawl's free 'AI Visibility Audit' guide shows how to check whether your site is in the crawl, whether CCBot is blocked, and whether CDN or firewall defaults are quietly blocking AI crawlers.
Meta's newest frontier model, Muse Spark (April 2026), is not open-weight. Llama 4 remains the latest Llama release.
The four layers
How Open-weight models builds an answer
Every AI answer is made in four layers: what the model learned in pre-training, how it was tuned, what it retrieves live, and how it puts the answer together. Here is each one for Open-weight models, and what it means for a brand that wants to be cited.
Layer 1
Pre-training: the base (foundation) model
This is the layer that matters most. Open-weight base models learn from huge web corpora frozen at a cut-off: Llama 4 Scout from about 40 trillion tokens of public, licensed and Meta product data across 200 languages; Gemma from web documents in more than 140 languages; Qwen3 from about 36 trillion tokens in 119 languages; DeepSeek-V4 from more than 32 trillion tokens; and Falcon from RefinedWeb, a filtered Common Crawl. Mistral discloses no data. Much of the web text reaches them through Common Crawl and the datasets filtered from it.
What it means for you: To be known by these models, a brand needs to be in the pages that survive filtering: crawlable without JavaScript, linked from well-connected sites, substantial, and consistent about the facts. This is what base model targeting works on, and why it takes a training cycle to show.
Layer 2
Post-training: tuning and alignment
Instruct and chat versions usually ship alongside the base weights. Reasoning versions are trained with reinforcement learning, as with DeepSeek-R1 and TII's Falcon-H1R, and several families can switch their thinking on and off. Licences generally let anyone fine-tune them further, and DeepSeek-R1's MIT licence explicitly allows distillation, so thousands of community variants exist.
What it means for you: Each fine-tune can blur or overwrite brand facts, and you cannot see most of them. That makes the base model's knowledge the common ground worth getting right: it is what every variant starts from.
Layer 3
Retrieval: live search
None in the weights. An open-weight model knows only what was in its training data up to the cut-off. Live information arrives only if the company deploying it adds search, retrieval from its own documents or tool calls, as with Mistral's web search tool or the built-in tools of Alibaba's hosted Qwen.
What it means for you: Where a deployer adds search, normal SEO and GEO apply through whichever search provider it uses. Where it does not, which is common in internal tools and offline apps, the model's memory is the only source and your website cannot correct it.
Layer 4
The answer: what people read
It depends entirely on the deployer. A raw open-weight model does not cite sources. Brand facts come from its memory and can be stale or blended with similar names, and quantised and fine-tuned versions multiply the number of slightly different answers in circulation.
What it means for you: You cannot buy placement or ask for a correction. The only way to change these answers is to change what the next generation of models learns from, and to measure the current ones directly.
Crawlers
Open-weight models's crawlers and user agents
The user-agent tokens to look for in your server logs and to name in robots.txt.
| User-agent token | What it does | robots.txt |
|---|---|---|
| CCBot Documentation | Common Crawl's crawler for its free monthly web archive, which many open-weight training datasets are built on. User agent: CCBot/2.0 (https://commoncrawl.org/faq/). It does not run JavaScript or use cookies, and backs off on 429 and 5xx responses. | Follows robots.txt (RFC 9309), Crawl-delay and nofollow, and reads sitemaps listed in robots.txt. |
| meta-externalagent Documentation | Meta's crawler for uses such as training its AI models and indexing content for its products: the relevant token for future Llama and Meta models. | Follows robots.txt, Meta says. Its user-requested fetcher, meta-externalfetcher, may not. |
| Google-Extended Documentation | Not a crawler: a robots.txt token controlling whether pages Google crawls may train future Gemini models and ground Gemini answers. Google does not say whether it covers Gemma. | A robots.txt control token, read by Google's existing crawlers. |
| MistralAI-Training Documentation | Builds training datasets for Mistral's models, including its open-weight releases. | Follows robots.txt, Mistral says. |
Alibaba (Qwen), DeepSeek and TII (Falcon) document no crawler of their own on their model pages. DeepSeek's EU training summary names Common Crawl among its sources, which makes CCBot the documented route into DeepSeek's training data as well.
Crawl sources
Where Open-weight models crawls from, and how to verify it
Common Crawl publishes CCBot's addresses as JSON at index.commoncrawl.org/ccbot.json. When we checked they were 2600:1f28:365:8000::/56, 3.41.188.32/29, 18.97.9.168/29, 18.97.14.80/29 and 18.97.14.88/30. They sit in Amazon's network (AS14618, according to RIPEstat), and the archive itself is stored in AWS's us-east-1 region.
To verify a request, use forward-confirmed reverse DNS: genuine IPv4 addresses resolve to hostnames such as 18-97-14-84.crawl.commoncrawl.org, which resolve back to the same address. IPv6 reverse DNS is not yet supported, so check IPv6 requests against the JSON list instead.
Meta publishes no IP list for meta-externalagent on its current crawler page, and Mistral publishes none for MistralAI-Training. Google-Extended uses Google's existing crawlers, so Google's own published ranges and verification steps apply. Alibaba, DeepSeek and TII publish no crawler IP information at all.
Opting out
How to opt out of Open-weight models, or limit what it uses
Block Common Crawl
User-agent: CCBot with Disallow: /. Common Crawl rechecks robots.txt periodically and the block stops future crawls, but past archives, the datasets built from them and models already trained are unchanged.
Official pageOr slow it down
User-agent: CCBot with Crawl-delay: 2 asks for two seconds between requests. CCBot also backs off by itself when it receives HTTP 429 or 5xx responses.
Official pageAsk Common Crawl to remove URLs
Email info@commoncrawl.org. Requests are logged in a public opt-out registry, and Common Crawl filters the URLs from later crawls and its public indices, but it cannot rewrite its archived files.
Official pageAsk the datasets built on it
Datasets filtered from Common Crawl handle removals separately. FineWeb, for example, has its own form for removing personal data.
Official pageRefuse named training crawlers
User-agent: meta-externalagent, Google-Extended or MistralAI-Training with Disallow: / refuses future training by Meta, Google's Gemini and Mistral respectively. None of them affects weights already released.
Check what was used
EU rules require providers of general-purpose AI models to publish a summary of their training content, including the top domains they scraped. Google and Mistral AI have signed the EU's GPAI Code of Practice; Meta, Alibaba, DeepSeek and TII were not listed as signatories when we checked.
Official page
Weights that have been downloaded cannot be recalled, so opting out shapes only future models. Blocking Common Crawl also removes you from the raw material of many models at once, including those whose answers you may want to be in. For most brands, being represented accurately is worth more than being absent.
How we help
How we get brands cited in Open-weight models
The Coaley Peak GEO Research Centre researches and implements GEO for SEO agencies and in-house teams, with foreign markets as our speciality.
Base model probing
We ask open-weight base models what they know about your brand, with retrieval switched off and across languages, to see what each one learned, what it gets wrong and which competitors it names instead.
Training-data visibility audit
We check whether your pages are in recent Common Crawl snapshots, whether CCBot or your CDN blocks them, and whether they render without JavaScript, then fix whatever keeps them out.
Content that survives filtering
Training datasets keep substantial, well-linked, clearly written pages and drop thin or boilerplate ones. We restructure key pages so the facts about you sit in the kind of text that is kept.
Crawler policy by purpose
We help you decide which training crawlers to allow and which to refuse, model family by model family, so that opting out of one use does not cost you visibility in another.
Tracking across model generations
Each new release resets what models know. We re-test as new Llama, Gemma, Qwen, DeepSeek, Mistral and Falcon versions appear, so you can see whether the work is reaching them.
More on our research, services and team at the Coaley Peak GEO Research Centre.
FAQ
Open-weight models GEO: questions
What is an open-weight model?
A language model whose trained weights are published for anyone to download and run, under a licence. Llama, Gemma, Qwen, DeepSeek, Mistral and Falcon are the best-known families. Unlike open-source software, the training data and code are usually not published.
How do open-weight models learn about a brand?
From their pre-training data, much of it web text filtered from Common Crawl, up to a cut-off date. Without search attached, a model can only repeat what was in that data, so brand facts published after the cut-off are unknown to it.
What is base model targeting?
Our specialist service for influencing what language models know from pre-training. We probe base and open-weight models with retrieval switched off, find where a brand is missing or misdescribed, and work on the sources future training draws on.
Does blocking CCBot remove my site from AI models?
No. It stops future Common Crawl captures, but past archives are immutable, and models already trained keep what they learned. It also removes you from the data many future models will learn from.
Which open-weight models publish a knowledge cut-off?
Llama 4 (August 2024), Llama 3.1 (December 2023), Gemma 4 (January 2025) and Gemma 3 (August 2024). Qwen and Falcon publish none, DeepSeek dates its data only in its EU training summaries, and Mistral gives dates only in its recommended system prompts.
How can I tell whether my site is in Common Crawl?
Search Common Crawl's public URL index for your domain, and look for CCBot from its published IP ranges in your server logs. Common Crawl's free AI Visibility Audit guide walks through both checks.
Want a free chat about Open-weight models?
Tell us which market and brand you are working on. If we can help, we will set up a free, no-obligation chat with Stephen or Ethan.
More engine guides
Related engines
Independence and trademarks. Llama is a trademark of Meta Platforms, Inc.; Gemma is a trademark of Google LLC; Qwen is a trademark of Alibaba Group or its affiliates; DeepSeek is a trademark of its respective owner; Mistral is a trademark of Mistral AI; Falcon is a trademark of the Technology Innovation Institute or its affiliates; Common Crawl is a trademark of the Common Crawl Foundation. All other trademarks are the property of their respective owners. Coaley Peak is independent: we are not associated with, endorsed by or a partner of Meta, Google, Alibaba, DeepSeek, Mistral AI, TII and the Common Crawl Foundation. This guide is our own research from public documentation and reputable reporting, checked in October 2026.
Reviewed by Stephen Grindley, research centre lead. Last checked .
Document reference: ISO_webpage_geo_engine_open_weight_models_v1
Last modified: 7 October 2026
GEO Research Centre·Engine guides·Open-weight models