Why AI Referrals Matter—and Why Blocking AI Crawlers Can Cost You Traffic

For years, website visibility followed a familiar pattern: publish useful content, make it accessible to search engines, earn rankings, and convert search visitors into customers. That model still matters, but a new discovery layer is growing alongside it. People increasingly ask AI assistants to recommend software, compare products, find service providers, explain complex topics, and identify businesses that match highly specific requirements.

When a person follows a link from an AI assistant to a company, article, or product page, the resulting visit is an AI referral. A mention without a click may influence later behavior, but it is not a directly attributable referral. For website owners, these referrals can become a meaningful source of awareness, qualified traffic, and new customers. Yet website owners can unintentionally limit this opportunity through broad bot-blocking rules, restrictive robots.txt directives, web application firewalls, or one-click settings that block every crawler labeled “AI.”

Protecting a website from abusive automation is sensible. Blocking all AI access without understanding the different types of crawlers, however, can also prevent legitimate search and assistant systems from discovering, refreshing, or retrieving public content. The better approach is not “allow everything” or “block everything.” It is to decide which uses support your business, then apply precise controls.

AI referrals are becoming part of the customer journey

Conventional search results typically present users with a list of links, while AI-assisted search experiences often synthesize information into a direct response. They may interpret the request, summarize relevant information, narrow the options, and recommend a small number of sources or businesses.

Consider the difference between these two searches:

“Email marketing software”
“What is the best email marketing platform for a five-person European retailer that needs multilingual automation and transparent pricing?”

The second request contains far more context and commercial intent. If an AI assistant recommends a suitable provider and the user follows the link, that visitor may arrive with a clearer understanding of the product and a stronger reason to evaluate it. This does not mean every AI-referred visitor will convert, or that AI traffic always performs better than search traffic. Performance varies by industry, assistant, query, and website. It does mean AI referrals deserve to be measured as a distinct discovery channel rather than dismissed as a novelty.

The direction of travel is already measurable. Adobe Digital Insights reported that referral traffic from AI sources to U.S. retail websites grew 393% year over year from January through March 2026. In March, AI-referred visits had a conversion rate 42% higher than non-AI traffic, spent 48% more time on site, and viewed 13% more pages per visit. Adobe’s findings cover its observed U.S. retail traffic, not every website or industry, but they demonstrate the commercial potential of a well-matched AI referral.

It is equally important not to inflate the channel. A 2026 Semrush study covering billions of visits across more than 50,000 websites found that AI referral traffic grew 66% during 2025 but still represented only about 0.14% of total visits in its dataset. In other words, AI referrals are strategically important because of their momentum and influence—not because they have already replaced traditional search.

AI visibility can also influence customers without producing an immediate click. A person may encounter a brand in an AI-generated comparison, remember the name, and visit directly later. Another may use an assistant for research before completing a purchase through search, email, or a sales conversation. The value of the channel may therefore extend beyond the referrals that appear neatly in an analytics report.

Crawlability creates eligibility, not a guarantee

There is an important limit to the argument for allowing AI crawlers: crawl access does not guarantee that an AI system will recommend, quote, cite, or link to a website.

For a page to have a strong chance of generating a current, accurately informed referral, several stages commonly matter:

A relevant search/index crawler or user-triggered fetcher must be permitted and technically able to access the page directly, although some systems may still discover its URL through third-party indexes or links.
It must discover or retrieve the relevant URL.
It must understand the content and connect it to the user’s question.
The content must be sufficiently useful, relevant, credible, and current.
The system must choose to mention or cite it.
The user must decide to visit the source.

Allowing a relevant search/index crawler or user-directed fetcher to access the page addresses only the first part of that chain. It can improve eligibility for discovery; it does not guarantee placement. Content quality, topical relevance, authority, freshness, technical accessibility, and the assistant’s own retrieval and ranking systems still matter.

The reverse is more straightforward: if a legitimate search crawler or user-directed fetcher is blocked, the system may be unable to inspect the page at all. That can reduce the site’s chance of consideration, leave the assistant with outdated information, or cause a user-requested retrieval to fail.

Not every AI crawler has the same purpose

The phrase “AI crawler” is often treated as though it describes one activity. In practice, providers operate different crawlers or control tokens for different purposes. The names and behaviors can change, so website owners should always consult each provider’s current documentation. The major categories are:

1. Model-training crawlers

Training crawlers collect public web material that may be used to develop or improve future AI models. Whether to permit this use is a policy, licensing, and commercial decision. Publishers with valuable proprietary archives may reach a different conclusion from a retailer that wants broad awareness of its public product catalog.

Allowing a training crawler should not be confused with joining an AI search index, and blocking one does not necessarily require blocking every search or assistant feature from the same company. For example, OpenAI documents GPTBot separately from OAI-SearchBot, while Anthropic distinguishes ClaudeBot from its search and user-directed bots. Google likewise provides the Google-Extended control token for certain Gemini uses and states that it does not affect inclusion or ranking in Google Search.

2. AI search and indexing crawlers

Search-oriented crawlers collect or refresh information so an AI product can find relevant sources when answering questions. These have the clearest relationship to potential discoverability, citations, and referrals.

OpenAI advises publishers who want their content considered for ChatGPT summaries and snippets not to block OAI-SearchBot. Anthropic explains that restricting Claude-SearchBot may reduce a site’s visibility and accuracy in user search results. These statements still do not promise inclusion, but they make the tradeoff clear: blocking search access can reduce the system’s ability to understand and surface a site.

3. User-triggered fetchers and agents

Some bots retrieve a page because a person has asked an assistant to visit, summarize, compare, or act on that page. Anthropic, for example, describes Claude-User as supporting retrieval initiated by Claude users. Cloudflare categorizes similar real-time activity as “Agent” behavior.

Blocking this category can affect a user at a moment of unusually high intent. A potential customer may explicitly ask an assistant to compare your pricing page, inspect your documentation, or summarize your product. If the request receives a firewall challenge or a 403 response, the assistant may be unable to complete the task and may rely on another source instead.

The hidden cost of a blanket block

Broad blocking rules can look attractive because they are simple. A website owner sees rising bot traffic, finds a control labeled “Block AI bots,” activates it, and assumes the job is complete. The unintended effects may be difficult to see because lost referrals never appear in analytics.

A blanket policy can create several kinds of opportunity cost:

Reduced discovery: Search-oriented systems may not be able to index important articles, product pages, documentation, or company information.
Stale answers: An assistant may rely on an older index, third-party descriptions, or competitors’ content because it cannot refresh your source material.
Missing citations: Even when your page is the original or most authoritative source, an inaccessible page is harder to verify and cite.
Failed user requests: An assistant acting at a user’s direction may be unable to retrieve a page the user specifically wants to evaluate.
Loss of high-intent visits: A site can miss customers who use conversational tools to assemble shortlists or make purchase decisions.
Inaccurate brand representation: If assistants cannot access your current pricing, features, policies, or documentation, they may produce incomplete answers based on secondary sources.

These costs are not proof that every crawler should be welcomed. They are reasons to understand what a rule blocks before enabling it.

Cloudflare offers controls—but the policy is still yours

Cloudflare and other security providers offer valuable defenses against scraping, denial-of-service attacks, credential abuse, and poorly behaved automation. Those protections can remain in place while crawler policies are refined. The risk comes from treating all automated requests as equivalent.

Cloudflare’s current AI bot policy documentation distinguishes among Search, Agent, and Training behavior. Its AI Crawl Control also lets site owners monitor activity and apply allow-or-block decisions to individual crawlers. Cloudflare itself notes that access may be appropriate for services that provide value through citations, referrals, or commercial agreements.

That granularity is useful. A site might choose to:

Allow reputable AI search crawlers on public editorial and product pages.
Allow user-triggered agents where they help customers research or use the site.
Make a separate decision about training crawlers based on licensing and content strategy.
Block unidentified scrapers, non-compliant bots, abusive request patterns, and crawlers that ignore published directives.
Apply tighter rules to expensive endpoints, account areas, checkout flows, internal search, or frequently changing inventory APIs.

Before activating a preset, review its current scope. Check whether it blocks only training activity or also covers search, mixed-purpose crawlers, and agents. Product defaults and classifications can change, so this should be a recurring review rather than a one-time configuration.

Build a policy around content and purpose

A sound crawler policy starts by classifying your own content. Public marketing pages, blog posts, help articles, store listings, and documentation exist to be discovered. Customer records, paid reports, private communities, staging environments, and account pages do not.

Do not rely on robots.txt to secure sensitive material. It is a crawl directive, not an authentication system. Private content should be protected with proper access controls. For public content, use provider-specific directives when you want to permit one purpose and decline another.

For example, a publisher may decide that AI search citations support its audience strategy while unrestricted model-training access does not. A software company may want assistants to retrieve public documentation but rate-limit expensive dynamic endpoints. An online retailer may welcome product discovery while excluding carts, customer accounts, and personalized pages. Precision allows each business model to make its own tradeoff.

Make public content easier to understand and recommend

Crawler access is only the foundation. Once a page is reachable, it still needs to communicate clearly. The same practices that improve search visibility and human usability also help AI systems interpret a site:

Use descriptive page titles and logical headings.
Answer important customer questions directly and in plain language.
Keep pricing, availability, specifications, policies, and contact details current.
Provide original facts, examples, research, or expert commentary rather than generic summaries.
Use stable canonical URLs and submit accurate XML sitemaps.
Add appropriate structured data without marking up information that users cannot see.
Include primary content in the initial HTML where practical, or ensure it can be rendered reliably without login requirements or user interaction.
Show authorship, update dates, company identity, and evidence where trust matters.
Link related pages so crawlers and visitors can understand the site’s subject coverage.

There is no need to rewrite every page for machines or chase speculative “AI optimization” tricks. Clear, factual, well-structured content that genuinely helps a defined audience is the more durable strategy.

Measure AI referrals instead of guessing

Website owners should evaluate AI visibility with the same discipline used for any other channel. Start by creating a baseline before changing crawler rules.

Track known AI referrers in web analytics, but remember that attribution may be incomplete when apps, privacy controls, copied links, or multi-step journeys remove referral information. Review landing pages, engagement, sign-ups, purchases, and assisted conversions. Add a “How did you hear about us?” field where appropriate, and include AI assistants as an option. Inspect CDN and server logs to see which verified crawlers request the site and whether they receive successful responses, redirects, rate limits, or blocks.

Hitsteps AI Referral Tracking can separate attributable visits from recognized assistants such as ChatGPT, Gemini, Claude, Copilot, and Perplexity, then connect the detected source with landing pages and available visitor-journey context. It is important to read that evidence correctly: not every assistant preserves a referrer or campaign signal, so detected AI referrals are a measurable subset rather than a complete count of every visit influenced by AI.

Then compare periods before and after policy changes. The goal is not to maximize crawler requests. It is to permit valuable discovery while controlling infrastructure cost, security risk, and unwanted content use.

A practical AI crawler audit checklist

Inventory every control layer. Review robots.txt, meta robots directives, CDN settings, WAF rules, bot-management presets, rate limits, CAPTCHAs, geo restrictions, and origin-server rules.
Check your actual public response. Confirm that important pages and robots.txt return the intended status code without authentication or JavaScript challenges.
Classify your content. Separate public discovery content from private, licensed, paid, transactional, or sensitive areas.
Classify crawler purposes. Distinguish training, search/indexing, and user-triggered retrieval using current provider documentation.
Avoid decisions based only on a bot’s company name. One provider may operate multiple bots with materially different functions.
Allow deliberately. Permit reputable search crawlers and user-directed fetchers where their access supports discovery and customer service.
Restrict deliberately. Block or limit training use, abusive bots, sensitive paths, or costly endpoints according to your business requirements.
Verify identity where possible. Use official IP verification, signed bot identity, or your CDN’s verified-bot data rather than trusting a user-agent string alone.
Prefer rate limits and path rules to unnecessary site-wide blocks. A narrow control can reduce load without making the entire public site invisible.
Monitor outcomes. Track crawler response codes, referral traffic, conversions, content freshness, and unexpected infrastructure costs.
Document the decision. Record who approved the policy, which crawler purposes it permits, and why.
Review regularly. Providers introduce new crawlers, names, controls, and classifications. Re-audit the policy at least quarterly and after major CDN changes.

Visibility requires a conscious choice

AI assistants are becoming another gateway between websites and potential customers. For many organizations, being accurately represented in that environment will matter alongside traditional search, social media, advertising, and direct traffic.

The right response is not unconditional access. Website owners have legitimate concerns about copyright, licensing, server load, competitive scraping, privacy, and security. They should retain control over how their content is used. But a single switch that blocks every AI-related bot can solve one concern while quietly creating another: reduced visibility where customers are increasingly asking for recommendations.

Treat training, search, and user-directed retrieval as separate decisions. Protect private and expensive resources properly. Keep public content accessible to the reputable systems that can help people find it. Measure the results, refine the policy, and remember the central principle: allowing a crawler does not guarantee an AI referral—but blocking the crawlers responsible for discovery or retrieval can prevent that opportunity before it begins.