Cloudflare blocking OAI-SearchBot? Diagnose and fix it.

1. Name the crawler and intended access

“Allow ChatGPT” is too vague for a configuration change. OpenAI identifies OAI-SearchBot as its search crawler, GPTBot as a crawler for content that may be used in training, and ChatGPT-User as a user agent for certain user-triggered visits. Search access and training preferences are separate. A successful ChatGPT-User visit does not establish access for OAI-SearchBot. See the OpenAI crawler reference.

Write a small access contract before touching the dashboard. For example: “Permit the search crawler to retrieve public product pages on www.example.com. Preserve our training preference and all account authorization.” Include the exact hostname and representative URLs. A working homepage says nothing about a challenged pricing page or a separate documentation subdomain.

Check the robots.txt served through that hostname, not just the copy in Git. Cloudflare can generate a managed file or prepend managed content to an existing file. Its managed robots.txt documentation explains this behavior. Read the complete delivered file for the applicable crawler group and path; do not remove a training restriction merely because it mentions AI.

Policy timing: checked on 12 September 2026, Cloudflare’s AI bot policy documentation announces new-domain defaults for 15 September: Training and Agent traffic blocked on pages with ads, with Search remaining allowed. Mixed Search/Training crawlers receive the training treatment. This is not a statement that every existing domain or every search crawler becomes blocked. Inspect your zone’s actual policy and recheck the documentation after rollout.

2. Capture the actual response

Start with one public URL you own or are authorized to assess. Use a GET request so you can inspect the document as well as the headers. Replace the example URL, and run from a directory where these evidence files will not overwrite something you need:

curl --location --compressed --max-time 30 \
  --dump-header page-headers.txt \
  --output page-body.html \
  --write-out 'Status: %{http_code}\nFinal URL: %{url_effective}\n' \
  'https://www.example.com/pricing/'

Repeat for /robots.txt and your sitemap using different output filenames. Keep the request time in UTC, final URL, and response files together. A redirect can move the request to another host with different settings; read every header block and identify the final response.

Cloudflare documents cf-mitigated: challenge as the marker for its Challenge Pages, which use an HTML content type. That is stronger evidence than guessing from a generic 403. See Detect a Challenge Page response. Save any cf-ray value to help correlate the request with Cloudflare events.

You can compare a second request using the current provider-documented user-agent string to detect user-agent-dependent behavior. Label it as a simulated probe and save it separately. It still originates from your machine. Curl does not automatically follow robots.txt policy or execute page JavaScript, and a passing probe cannot test OpenAI’s network identity.

If the response contains only an application shell, continue with the AI crawler JavaScript rendering audit. Changing firewall rules will not put missing product facts into the HTML.

3. Verify crawler identity and locate the event

A user-agent field is a claim made by the client. Cloudflare says the Free plan’s AI Crawl Control detection uses user-agent strings; stronger detection is available with other capabilities. A named row in that dashboard therefore needs interpretation, not automatic trust. Read its detection and plan guidance.

For a candidate OAI-SearchBot request, compare the edge-observed connecting IP against OpenAI’s current published searchbot CIDR ranges. Use a real IP/CIDR membership check, including the address family involved, rather than matching the first few digits. Keep the list retrieval time and result with the request record. Do not copy a permanent list of addresses from a blog post.

Use client identity from a trusted logging boundary. An origin behind a reverse proxy may see the proxy’s connection address, while an arbitrary forwarded header can be supplied by a caller if your trust configuration permits it. Have the infrastructure owner identify which field represents the client at the edge before making an exception.

In Cloudflare’s Security analytics, open Events and narrow the time window around the failed request. Filter using available request fields and inspect the event details for the action, security service, rule identifier, host, path, and client identity. Cloudflare’s Security Events documentation notes that results may be sampled and that one request can generate multiple events. No matching row is therefore not proof that nothing was blocked.

Our recommended evidence labels are probe only, identity supported by current provider ranges, and identity unknown. Keep the latter when you lack edge logs. This makes a report useful without quietly converting a guessed identity into a security decision.

4. Change the control responsible for the failure

Choose the remedy from the event’s service and rule, rather than cycling through settings until a test passes. Save the original configuration, the intended public scope, and a rollback condition.

AI Crawl Control says Allow, but another rule blocks

Cloudflare implements crawler blocks through WAF custom rules. An allowed crawler is not covered by that block selection, but earlier custom rules can still deny it, and those denials may be absent from AI Crawl Control analytics. Review the matching rule and its order. Cloudflare describes this explicitly in AI Crawl Control with WAF.

If a broad country or bot rule is responsible, make the smallest supported adjustment consistent with your access contract. A rule review should identify the trusted crawler identity, exact public host and paths, permitted retrieval methods, and the specific control being excepted. A substring match on “OAI-SearchBot” is not an identity check.

The service is Bot Fight Mode

A WAF custom Skip rule cannot bypass Bot Fight Mode. Cloudflare’s Bot Fight Mode documentation distinguishes it from Super Bot Fight Mode, which supports scoped Skip rules. If Bot Fight Mode caused the false positive, decide whether to change that feature or use a control that supports the exception you need. Evaluate the site-wide effect before disabling a protection.

The service supports a scoped exception

For products that support it, Cloudflare’s Skip rule configuration lets you choose the features to skip. Review those selections explicitly. A rule intended to avoid one false positive should not silently bypass all managed protections, unrelated custom rules, and rate limits.

Keep account pages, administrative routes, and state-changing requests outside the public-content exception. Confirm authorization still runs at the application layer. Our WebMCP security checklist covers permissions for agent actions; crawler access itself should grant no account privileges.

The request passes WAF but still fails

Cloudflare documents bot protections after custom rules and pay-per-crawl after bot protections. Check later controls and the origin response instead of assuming the first allowed layer settles delivery. See AI Crawl Control with bot solutions. If you use paid crawl access, include that policy in the investigation rather than misclassifying it as an accidental block.

A fictional pricing-page failure, worked through

Consider an imaginary software company whose public pricing URL works in a signed-in employee’s browser. Its deployed robots.txt permits OAI-SearchBot, and AI Crawl Control displays Allow. A local probe receives a challenge. Those observations establish an access discrepancy, but not yet a blocked OpenAI visit.

The operator locates a separate request carrying the crawler name. Its edge client IP falls within the current provider-published range, and the event identifies a custom rule named “Challenge automated traffic.” The rule covers the entire hostname. The origin has no corresponding application request, consistent with a request stopped before it reached the application.

The team changes that rule’s coverage for the approved crawler’s public documentation and pricing paths using identity criteria supported by its configuration. It keeps the account and checkout routes protected. It records the prior rule expression so the change can be reversed.

After deployment, the local probe and a later provider-range-matching crawler request receive the expected public response. An unauthorized account request remains denied. The report can now say that the observed crawler request succeeded after the rule change. It cannot say that ChatGPT indexed the page, selected it for an answer, or sent a customer.

This is an editorial scenario showing how to reason from evidence. It is not an experiment, customer result, or claim about Adapt My Page’s infrastructure.

5. Verify the fix at three different levels

Configuration and public response

Fetch the delivered robots.txt, sitemap, and target page again. Confirm the intended host, final status, content type, and expected facts. Compare the policy with your written contract. Exercise a representative excluded route with a test account or signed-out request you are authorized to use; confirm it still enforces the intended boundary.

Observed crawler delivery

Review a later request whose identity is supported by your verification method. Record its time, response outcome, and applicable rule evidence. If none arrives during your observation window, report “configuration corrected; actual crawler revisit not yet observed.” A crawler cannot be made to revisit simply by renaming your probe.

OpenAI notes that robots.txt changes can take about 24 hours to be reflected in its systems. That is a policy-update estimate, not a guaranteed crawl, indexing, or citation deadline. Keep these outcomes separate when reporting to a stakeholder.

Discovery and useful content

Once delivery works, check that public pages are linked, canonical URLs are consistent, and visible answers are accurate. Use the main checklist’s discovery checks. An accessible page can still be irrelevant to a query, stale, or too ambiguous to support a useful answer.

• The crawler, host, paths, and intended use are written down.
• The deployed policy and responsible security rule are recorded.
• Response files contain the intended public facts, not an interstitial.
• The identity method and observation window are explicit.
• Private routes and unauthorized actions remain protected.
• The result distinguishes a passing probe from an observed crawler visit.

When your hosting provider controls Cloudflare

If a managed platform owns the zone, prepare a request it can investigate. Send the public hostname and path, UTC time, status, challenge marker if present, Ray ID if available, and the intended crawler policy. Explain whether the request was your probe or an observed crawler visit. Ask which service and rule produced the response and whether it supports an exception scoped to your public content.

Keep credentials, session cookies, and customer data out of the handoff. If the platform cannot expose or change the responsible control, record that exact limitation. Adding llms.txt or rewriting page copy cannot repair a delivery restriction outside your control.