To find out whether answer engines have your content, check four places that Search Console does not cover: your robots.txt file, your firewall rules, your server logs, and your referral data. Search Console reports on Google. ChatGPT, Perplexity, Claude and Copilot run their own crawlers and keep their own indexes. We know of no per-site report from those operators as of this writing. A blank dashboard means the data is missing. It does not mean your content is absent.
This article walks through the audit using one example from wealth management. It also explains what each check can and cannot prove.
What Search Console Does and Does Not Cover
Search Console has changed this year. Its generative AI performance report shows how your site performs in generative AI features on Google Search. Google says it rolled these insights out to all websites worldwide as of August 31, 2026. The report includes data from the Web search type in the Performance report. So it measures Google's own features and nothing else.
Two other details matter for an audit.
- Google-Extended is a control, not a crawler. Google describes it as a standalone product token that publishers can use to manage whether crawled content may be used for training Gemini models and for grounding in Gemini Apps. Crawling uses existing Google user agent strings. You will not see "Google-Extended" in your logs.
- Googlebot in your logs means Google Search. Google's engineers put it this way: when you see Googlebot in your server logs, you are looking at Google Search.
Neither tells you anything about other assistants. Those need their own checks.
Crawled, Indexed, and Cited Are Three Different States
A page can be in one state without the next.
- Crawled: a bot requested the page and received it.
- Indexed: the engine stored the content in a form it can retrieve later.
- Cited: the engine used the page in an answer and may have linked to it.
Some assistants also fetch a page live when a user asks about it. OpenAI, for example, documents separate user agents for training, for search, and for user-requested fetches. Operators change these names. Read each operator's current documentation before you build a list.
Your own data can show crawling directly. Indexing and citation are harder to see, and the checks below only estimate them.
A Worked Example: One Registered Investment Adviser's Audit
Dana is a composite, an invented advisor, not a client of ours. She runs a four-person registered investment adviser (RIA) firm. It publishes a monthly retirement-income commentary and a quarterly market letter as a PDF. Search Console shows steady impressions. Then a prospect says she asked an AI assistant about Dana's firm and the assistant found nothing. Dana wants to know whether the firm's content is reaching answer engines at all.
Step 1: Read the Robots.txt File by Crawler
A robots.txt file sits at the root of the site. Unless you say otherwise, all files are implicitly allowed for crawling. The catch is how rules are matched. A user agent can match only one rule set, the first, most specific group that matches.
Dana's web vendor once added a group for one AI crawler. That group allows the whole site. The general group for all crawlers still blocks the folder that holds the market letters. The named crawler follows its own group only, so it reaches that folder. Any other AI crawler follows the general group and is kept out. The file looks fine at a glance. It treats each crawler differently.
Remember too that robots.txt is a request. The instructions cannot enforce crawler behavior. It is up to the crawler to obey them.
Step 2: Check Firewall and Bot-Protection Rules
A robots.txt file can allow a crawler while the server refuses it. Some content delivery networks (CDNs) and web application firewalls offer bot-protection settings. These can return an error to bots they do not recognize. Googlebot is usually recognized and allowed. Search Console would therefore look healthy while another crawler gets refused.
Dana asks her host for the bot-protection settings and any rules that block automated traffic. Then she checks that the response code for each crawler is a normal success.
Step 3: Filter the Server Logs
Server logs record every request, including those from bots. Dana filters for the user agents each operator documents. She counts requests per crawler, per page, per month.
A user agent string can be faked, so she verifies the source. Google, for example, publishes the Internet Protocol (IP) address ranges its common crawlers use. It also documents a reverse lookup in the Domain Name System (DNS), the system that links an address to a host name. The host name for a genuine Google crawler matches a set pattern. Other operators publish their own IP lists. Dana matches requests against them.
Logs prove that a bot fetched a page. They do not prove it was indexed or cited.
Step 4: Look at What the Crawler Receives
Dana loads the commentary page without running scripts and checks whether the article text is in the first response. If the text only appears after JavaScript runs, a crawler that does not run scripts will see an empty shell. Crawlers differ in how they handle scripts. Check each operator's documentation for its behavior. Where the documentation is silent, treat this as a risk to test, not a fact.
Page size is another limit. For Google's crawler, Google states that bytes after a 2MB threshold are not fetched, rendered, or indexed. Other crawlers may set different limits. Dana's market letter PDF is small, but a long page with heavy embedded code can approach it.
Step 5: Read Referral Traffic Carefully
Google Analytics has an AI Assistant channel. It covers visits from sources like ChatGPT, Gemini, DeepSeek, Copilot, or Grok. It excludes Google's AI Overviews and AI Mode. The channel is assigned when the referrer matches a list of AI assistants.
A visit appears only when someone clicks a link and the referrer is passed along. A citation that nobody clicks leaves no referral. A zero here is ambiguous.
Step 6: Test with Repeated Prompts
Dana asks several assistants the questions a prospect might ask. One example is "How do fee-only advisors in her city explain retirement-income planning?" She records whether her firm appears or is cited. Answers vary between runs, so this is sampling, not measurement. Dana repeats it on a schedule and keeps the same prompts.
A Record Template for the Audit
Dana keeps one record per crawler per review.
- Crawler name: [user agent token from the operator's documentation]
- Operator's stated purpose: [training, search, or user-requested fetch]
- Allowed in robots.txt: [yes or no, and which rule group applies]
- Firewall response: [response code returned to a verified request]
- Requests in logs: [count for the review period and the pages requested]
- Text present in first response: [yes or no, per key page]
- Referral visits: [count from the AI Assistant channel]
- Prompt test result: [prompts used, date, whether the firm was cited]
- Reviewer: [name and title]
- Date reviewed: [date]
What the Evidence Can and Cannot Support
Each check supports a different conclusion.
- Logs with verified requests: the crawler can reach the page.
- Blocks in robots.txt or the firewall: the crawler probably cannot read the page. Fix these first.
- Referral visits: at least one assistant linked to the page.
- Prompt results: a sample of what assistants say today. They are not a ranking.
- No data anywhere: the cause is unknown. Check access, then check the content itself.
None of these checks can promise a citation. Engines choose sources by their own signals, and they do not publish all of them. What the audit can do is remove the avoidable causes. In Dana's case, that means an inconsistent robots.txt file and a firewall rule. Those are problems a firm can find and fix.
