Illustrative photograph. Photo: Rear of rack at NERSC data center - closeup.jpg by Derrick Coetzee, via Wikimedia Commons — CC0. Downloaded Wikimedia version; no custom edits.
Your homepage works in your browser. That does not establish what a search crawler receives.
A content delivery network may challenge unfamiliar requests. A page may return a successful status while showing an error screen. Important service information may be absent from the delivered document.
Before buying another visibility report, ask your website owner to document what the site actually serves.
Distinguish search access from training permission
OpenAI’s documentation identifies OAI-SearchBot for search and GPTBot for potential model-training collection. Their controls are independent. ChatGPT-User handles some user-initiated visits and has a different role. OpenAI also publishes crawler IP ranges. See the crawler reference.
Make an explicit decision for each purpose. Do not change all bot rules as a blanket fix, and do not assume a training opt-out is automatically a search opt-out.
Check representative pages
Test the homepage, a key service or product page, a pricing or policy page, and a location page if relevant.
For each, capture:
The requested URL and final destination.
The HTTP response status and relevant headers.
Whether the response contains the expected content.
Whether access requires a login or browser challenge.
Whether essential facts are available as readable text.
Run tests through the same public domain buyers use. A staging site or internal preview does not establish the behavior of the live site.
Inspect the layers that can block a request
Review robots rules alongside CDN settings, firewall rules, rate limits, hosting protections, and application behavior.
Changing a user-agent string in a test request is useful for inspecting a response, but does not prove that a request came from a genuine OpenAI crawler. For real log analysis, use the current published verification information rather than trusting a claimed name alone.
Avoid disabling broad security protections to solve one access issue. Identify the specific rule and make the narrow change appropriate to your site.
Check the content, not just the status code
A successful response can still contain an empty shell, an outdated page, or a message asking the visitor to enable something.
Ask whether the public response explains your offer and the critical conditions. Where your site relies heavily on scripts, verify delivery rather than assuming either universal success or universal failure.
Also inspect redirects, duplicate destinations, indexing directives, and links to important pages. Record what each check proves. A working URL says something about access, not recommendation quality.
Preserve a small evidence pack
Save the test date, page list, responses, relevant configuration, and changes made. Retest after a deployment that affects routing, rendering, or bot protection.
If you see a verified crawler visit in your logs, count it as a visit by that crawler. It is not a customer impression, a citation, or an endorsement.
Once access is established, move to the next question: does the available content accurately answer a buyer’s needs? Technical access and useful information require separate checks, and both need owners.
Next read: Turn product claims into buyer evidence.
