If you run a business on Meta’s platforms, Meta may soon have permission to crawl your website and use what it finds to train its AI. And no, you do not need to be an advertiser. Running a Facebook Page or opening Ads Manager is enough.
Meta has rewritten its Commercial Terms, with the new version taking effect on October 30. The update contains a crawling clause that lets Meta, or anyone acting on its behalf, send scrapers and automated tools across the websites connected to your business and use the collected content to develop, improve, research, and train its products. This is a meaningful expansion of rights the company previously reserved for advertisers. Now it reaches every business user of Meta’s products.
The news broke when Nieman Lab reported the change on October 15, after three legal professionals reviewed the terms for the outlet. Meta spokesperson Andy Stone responded on X the next day, calling the summary false and saying the update is meant to ensure Meta can pull accurate product information from third-party websites for ad and content recommendations, adding that it is not about scraping news sites. The two sides are reading the same document very differently, which is exactly why marketers need to understand what it says.
What Actually Changed
Meta’s current Commercial Terms, dated March 6, 2024, contain no crawling clause at all. The crawling permissions lived in Meta’s advertising terms, where placing an ad order granted Meta the right to crawl the advertiser’s domains, pages, and URLs, with uses that included developing products, maintaining integrity, and “research and training.”
The October version reaches much further. The new paragraph gives Meta permission to use web scrapers, bots, and other automated tools to crawl and collect content from domains and pages associated with a business’s products or services. It also covers URLs made available through a business’s use of Meta’s products, or through users’ use of those products. Notably, the clause lets Meta create test credentials to reach those pages, which suggests the door can open to content sitting behind registration walls.
The same crawl permission appears in the revised Self-Serve Ad Terms, also effective October 30. And there is a quiet line with a hard deadline attached: Meta considers your continued use of its services for business purposes to mean you have agreed to the new terms. If you disagree, the terms say you may stop using Meta’s products for business purposes. There is no accept or decline button. Simply staying is the acceptance.
Why This Is Bigger Than It Looks
The spokesperson’s explanation is plausible as far as it goes. Meta genuinely benefits from accurate product information on third-party websites, and better product data means better ad and content recommendations for users. That motivation is probably real.
But the permission is not written narrowly around product data. It covers any content from domains and pages associated with a business’s products or services, plus URLs surfaced through how the business or its users interact with Meta’s products. The legal professionals who reviewed the terms for Nieman Lab said this language could support an argument by Meta that it is permitted to scrape websites of news sites and use that content to train its AI products. Media and IP attorney Matt Topic said that while no one can be sure a court would enforce the provision, publishers should assume it will.
The shift from advertisers to everyone is the real story. Before, an advertiser handing Meta their domain URLs was trading crawl access for ad services in a fairly legible bargain. Now the grant attaches to any business use of Meta’s products, including brands that simply maintain a Facebook Page without buying a single ad. That is a far larger population, and most of them have no idea these terms are about to apply to them.
It also widens Meta’s AI training pipeline. The company already runs a family of crawlers: Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products through direct indexing, while Meta-WebIndexer improves Meta AI search results. Training data is the fuel of the current AI race, and terms of service that grant crawling and training rights across Meta’s entire business user base are a significant asset in that race, whatever the stated motivation.
The reaction from publishers reflects that concern. The European Publishers Council is writing to the European Commission about the new terms, worried that Meta could claim permission to use an article when a user shares a link to that story via social media. A reader sharing your article on Facebook turning into Meta claiming rights to crawl it shows how broadly this clause could reach in practice.
The Robots.txt Question Nobody Can Answer Cleanly
Meta’s spokesperson said the new terms will not change how the company crawls publishers, and that content owners can still manage access with robots.txt. The crawler documentation describes a layered setup: you can block Meta-ExternalAgent, the AI training crawler, while allowing Meta-WebIndexer, the search crawler whose inclusion helps Meta cite and link to your content in Meta AI responses.
Two problems keep this from being a settled answer. First, neither the updated Commercial Terms nor the updated Self-Serve Ad Terms mention robots.txt at all. The experts who reviewed the terms said the new language gives Meta an opening to argue that its commercial terms supersede other blocking instructions. Meta declined to answer whether test credentials could bypass paywalls or other paid content.
Second, the test credentials clause complicates everything. Cornell law professor James Grimmelmann noted that test credentials could technically let Meta sign up as a fake user to get past registration walls. A robots.txt file is a request that polite crawlers honor; it is not a lock. When a contract permits creating credentials to reach gated pages, the robots.txt conversation stops being about a simple on or off switch.
None of this is a reason to panic, but it is a reason to retire the old mental model. The old model said Meta crawls your site if you run ads, and robots.txt governs the details. The new model says Meta may crawl your site because you have a Facebook Page, may use what it collects to train AI, and may create credentials to reach pages robots.txt cannot see. That is a different bargain, and it arrives by default on October 30.
What Marketers Should Do Before October 30
Start by mapping what Meta can see. List the domains and pages associated with your products or services, including the ones Meta’s products surface: product catalog URLs, landing pages linked from ads or shops, and pages shared through your Facebook Page. Ask yourself what content on those pages you would not hand over for AI training if someone asked you directly. Your original research, customer stories, and gated guides: decide which of these are acceptable to share and which are not.
Next, audit your robots.txt file against Meta’s crawler documentation. Check how you currently treat Meta-ExternalAgent, Meta-WebIndexer, and Meta-ExternalFetcher. A blanket allow or disallow may be giving away more or less than you intend. Remember that robots.txt is only one layer, and consider stronger controls on the pages that matter most, such as requiring real authenticated accounts for premium content.
Then, treat this as a legal question, not just a technical one. If your business produces original content with real value, have counsel review the new Commercial Terms and Self-Serve Ad Terms before October 30. Only your lawyer can tell you what the continued-use clause means for your specific content and jurisdiction.
Finally, make the strategic call. For some businesses, the trade favors you: better product data means better recommendations, more visibility, and more sales. For publishers and businesses whose content is their core product, the calculation looks different. The one thing nobody should do is let October 30 pass without noticing. Continued use is consent, and silence is a choice.

FAQ
Do Meta’s new terms apply to me if I never run ads?
Yes, if you use Meta’s products for business. The new Commercial Terms tie the crawling grant to any business use, from running ads to managing a Facebook Page. Advertisers were already covered under the ad terms; this update brings non-advertisers into the same arrangement.
What exactly can Meta crawl?
Meta, or anyone acting on its behalf, may use scrapers, bots, and automated tools to crawl and collect content from domains and pages associated with your products or services, including URLs made available through your use or your users’ use of Meta’s products. The clause also permits creating test credentials to reach those pages.
Can Meta use my content to train AI?
The terms list training among the permitted uses, alongside developing, improving, and researching Meta’s products. Meta’s spokesperson says the update is about accurate product information, not scraping news sites, but the contract language is broader than that stated purpose.
Does robots.txt still control Meta’s crawling?
Meta points to robots.txt as the control mechanism, and its documentation shows how to block individual crawlers like Meta-ExternalAgent. But the new terms never mention robots.txt, legal experts say the contract language could be read to supersede it, and Meta declined to say whether test credentials could bypass paywalls.
What happens if I do nothing before October 30?
Continued business use of Meta’s products is treated as agreement to the new terms. If you disagree, the terms say you may stop using Meta’s products for business purposes. There is no separate opt-out.
Should small businesses worry about this?
It depends on what your content is worth. If your website is mostly a brochure and a shop, the trade may favor you. If you produce original research, journalism, or proprietary content, get legal advice before the terms take effect. Either way, review your robots.txt and know what Meta can reach.