On this page
- How do AI assistants find the products they cite?
- Which crawlers need access, and which can you block?
- What makes a software page easy to cite?
- Why do third-party pages carry so much weight?
- Do you need an llms.txt file?
- How do you measure AI referrals and citations?
- What should a small software team do first?
To get a software product cited by ChatGPT, Perplexity, Google's AI Overviews and AI Mode, Claude or Microsoft Copilot, two things have to be true. The search systems those assistants query while they answer must be able to reach and index your pages, and those pages must say plainly what the product is, what it costs and how it compares. In practice that means five jobs: let the search crawlers in, get indexed by Google and Bing, write pages that answer buyer questions with specific and dated facts, earn mentions on third-party pages people already trust, and measure what comes back.
This guide is for small teams behind SaaS products, developer tools and open-source projects. The crawler rules and product behaviors below were checked against each vendor's own documentation in November 2026. We publish iThub, a software directory, and we say so where it comes up.
How do AI assistants find the products they cite?
An assistant can know about your product in two ways. The first is training data, the text a model saw before its cutoff: months old by the time the model ships, impossible for you to correct, and rarely accompanied by a link. The second is retrieval: the assistant runs searches while it writes the answer, reads a handful of pages and cites them. Prices, versions, comparisons and anything launched recently come from retrieval, and so do nearly all clickable citations. Retrieval is the part you can influence.
Each assistant retrieves from a different place:
- ChatGPT. OpenAI's help center says ChatGPT search rewrites a question into one or more targeted queries and sometimes sends them to third-party search providers, and it names Microsoft among them (OpenAI help center). OpenAI also runs its own search crawler, OAI-SearchBot, and a site has to allow it to be eligible for ChatGPT search results.
- Google AI Overviews and AI Mode. Both draw on Google's own index and core ranking systems. Google's guide to generative AI features names two techniques: retrieval-augmented generation, and query fan-out, where the model issues several related searches to cover subtopics. A page must be indexed and eligible for a snippet to be shown.
- Microsoft Copilot. Copilot grounds its answers in Bing, and Bing Webmaster Tools now reports the "grounding queries" Copilot used when it cited a page (Bing Webmaster Blog).
- Perplexity. Perplexity crawls with its own bot, PerplexityBot, which it describes as the crawler that surfaces and links websites in Perplexity's results (Perplexity docs).
- Claude. Anthropic runs Claude-SearchBot to index pages for search quality and Claude-User to fetch pages when someone asks Claude a question (Anthropic).
Fan-out has a practical consequence. In an Ahrefs study published in March 2026, covering 863,000 keyword results and 4 million cited URLs, only 37.9% of the pages cited in AI Overviews ranked in Google's top 10 for the query that triggered the overview, and 31% did not rank in the top 100 at all (Ahrefs). Ahrefs reads this as AI Overviews pulling from the results of the related queries they generate. For a software product those related queries are predictable: "[product] pricing", "[product] vs [competitor]", "[product] self-hosted", "[product] alternatives", "[product] API". A page that answers one of them well can be cited even if it never ranks for the head term.
For planning purposes: proper indexing in Google and Bing covers Google's AI features, Copilot and part of ChatGPT search. Allowing three vendor search crawlers covers most of the rest.
Which crawlers need access, and which can you block?
AI companies now split their bots by job: one collects training data, one builds a search index, and one fetches a page when a user asks for it. The split matters because you can refuse training without disappearing from answers.
| Assistant | Where cited pages come from | User agent to allow | How to measure |
|---|---|---|---|
| ChatGPT search | OpenAI's own crawl plus third-party search providers, including Microsoft | OAI-SearchBot | Referrer chatgpt.com, utm_source=chatgpt.com |
| Google AI Overviews and AI Mode | Google Search index | Googlebot, with the Search generative AI setting left on "include" | Search Console Generative AI performance report |
| Gemini app | Google Search index, used for grounding | Googlebot, and do not block Google-Extended | Referrer gemini.google.com |
| Microsoft Copilot | Bing index | Bingbot | Bing Webmaster Tools AI Performance, referrer copilot.microsoft.com |
| Perplexity | Perplexity's own crawl | PerplexityBot | Referrer perplexity.ai |
| Claude | Claude's web search | Claude-SearchBot and Claude-User | Referrer claude.ai |
The training tokens are separate. OpenAI says disallowing GPTBot signals that content should not be used to train its models, while OAI-SearchBot controls search inclusion, and the two settings are independent (OpenAI crawler docs). Anthropic's equivalent is ClaudeBot. Google-Extended is a robots.txt token with no user agent of its own; Google says it does not affect inclusion or ranking in Google Search, but it also governs grounding in the Gemini app and Vertex AI, not only training (Google's crawler list). Blocking it therefore costs you Gemini app answers. Perplexity says PerplexityBot is not used to collect content for AI foundation models.
User-initiated fetchers behave differently. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person triggered the request, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic lets site owners control Claude-User in robots.txt and warns that blocking it may reduce visibility in user-directed search.
A robots.txt that stays in AI answers but declines training looks like this:
# Search and answer engines
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Model training (blocking Google-Extended also removes Gemini app grounding)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Googlebot and Bingbot fall under your normal rules. Two other things can block access without anyone noticing:
- CDN and firewall bot rules. OpenAI asks sites to allow requests from its published IP ranges, and Anthropic warns that IP-based blocking may not work reliably. If your CDN has a one-click setting for blocking AI bots, check whether it groups search crawlers with training crawlers.
- Google's new Search Console setting. Since August 31, 2026, every site has a Search generative AI control under Settings in Search Console. "Include" is the default. Choosing "exclude" removes your links and content from AI Overviews, AI Mode and generative features in Discover, and it does not change model training, which stays with Google-Extended (Search Console Help). Confirm nobody on the team has flipped it.
Training opt-outs are also moving toward a shared signal from the IETF's AI preferences (aipref) working group. Anthropic says it began recognizing that signal on October 30, 2026, with rules that name ClaudeBot taking priority. Google says it intends to support it in early 2027.
What makes a software page easy to cite?
An assistant quotes a passage that answers the question, names the product and states a fact it can attribute. Google says you do not need to break content into fragments or rewrite it for its AI systems, and none of the practices below require that.
Name the product and its category in the first sentence. "Acme Sync is an open-source file sync server for teams that self-host" can be quoted. "The fastest way to keep your team in sync" cannot. Do this on the homepage, the pricing page, the docs landing page and the README, and spell the name the same way everywhere, so "AcmeSync" on GitHub and "Acme Sync" on the site do not read as two products.
Put the answer directly under the heading. A docs page titled "Does Acme Sync support S3?" should say yes or no, and from which version, in its first line. Context and caveats come after.
State facts with dates. Pricing in a currency, with the billing period, what the free plan includes and an "as of" date. Supported platforms with version numbers. Release notes with release dates. A visible "last updated" date on docs pages. A date tells an assistant, and a reader, which of two sources is current.
Publish comparisons a competitor would call fair. "Best X for Y" and "X vs Y" are the questions where assistants name products. A comparison page with a criteria table, version numbers and the rows where you lose is more citable than a page that wins every row. Keep the list of comparisons short and real. Google's guide warns that producing separate pages for every query variation, fan-out queries included, to manipulate AI responses violates its scaled content abuse policy.
Serve the facts as HTML. Google says it can process JavaScript as long as it is not blocked. Other vendors do not document how their fetchers handle it, so render pricing, feature lists and docs on the server. A quick test: fetch the page with curl and search the raw HTML for your price.
Add structured data where it helps. Organization markup with sameAs links to your GitHub, registry and social profiles ties the entity together, and SoftwareApplication markup can carry applicationCategory, operatingSystem and offers. Google is explicit that structured data is not required for its generative AI features and that no special markup exists for them. Use it to keep facts consistent, and make it match the visible page.
Follow Bing's checklist. When it launched its AI Performance report, Bing listed what helps content get cited: clear headings, tables and FAQ sections, claims supported by data and sources, content kept current, and IndexNow to tell search engines when a page changes.
Why do third-party pages carry so much weight?
Ask an assistant for "the best self-hosted alternative to X" and it looks for agreement across sources. Your own site is one source, and an interested one. Google's guide notes that its AI features can show what is said about products in blogs, videos and forum discussions, and that chasing inauthentic mentions does not help. Our guide on whether directory submission is worth it covers the research on mentions and AI visibility in more detail. The places that matter for software:
- The GitHub README. For an open-source project it is the canonical description. Make the first paragraph a definition, then cover license, install and a link to the docs, and set repository topics.
- Package registries and marketplaces. Npm, PyPI, crates.io, Docker Hub and editor marketplaces all have description and homepage fields that get indexed. Fill them in with the same one-line definition.
- Review platforms. For B2B SaaS, the review sites your buyers already read. Claim existing profiles and correct outdated pricing.
- Curated directories. Directories that review submissions and publish structured facts, not link farms. A high Domain Rating alone does not tell you which is which. iThub is ours: listings are checked against the vendor's website, documentation and repository before publication, and profile pages are served as plain HTML with SoftwareApplication structured data (we checked the source of the n8n profile while writing this). The methodology page explains the review, and a free listing is available through Submit a tool.
- Community answers. Useful replies in GitHub Discussions, Stack Overflow, Reddit or Hacker News threads where your product genuinely fits, with your affiliation stated.
- Video. YouTube is the most-cited domain in AI Overviews in Ahrefs' data and accounted for 5.6% of all cited URLs in its March 2026 study. A short demo with a descriptive title and an accurate transcript gives assistants a second, independent source.
- Wikipedia and Wikidata, only if the product is notable. Wikipedia requires significant coverage in independent reliable sources and treats writing about your own company as a conflict of interest. Earn the coverage first.
Do you need an llms.txt file?
llms.txt is a proposed Markdown file at the root of a site that lists its most useful pages for language models. Google's position is explicit: Google Search does not use llms.txt files, and creating one neither helps nor harms visibility in Search, including its AI features (Google).
Usage data from elsewhere is thin. Ahrefs looked at the 137,210 domains in its web analytics product with traffic in May 2026: 28% published an llms.txt, which Ahrefs treats as an upper bound because its customers skew technical, and 97% of the valid files received zero requests that month. Of the requests that did arrive, 96% came from bots, and named AI tools made 19.5% of fetches (Ahrefs).
If your docs are already generated as Markdown, an llms.txt costs little to maintain and can help a developer who pastes your docs into a coding assistant. Do not count it as a citation strategy. If your product has an API, an MCP server is a more direct way to let agents use it, which is a different goal from being cited.
How do you measure AI referrals and citations?
No single report covers every assistant, so combine five partial views.
1. Referrals in analytics. The assistant domains are chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com and claude.ai. In GA4, create an exploration or a custom channel group that matches session source against a pattern such as:
chatgpt\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai
ChatGPT also appends utm_source=chatgpt.com to the links in its answers. That tag is not guaranteed: in October 2026 a bug removed it for about a week, links opened with neither tag nor referrer, and analytics tools recorded those visits as direct traffic until the tag returned (Search Engine Roundtable). Treat AI referral counts as a floor, and look twice at unexplained rises in direct traffic to deep pages.
2. Google Search Console. Clicks from AI Overviews and AI Mode arrive as ordinary Google organic traffic and cannot be separated in analytics. Search Console's Generative AI performance report, available to all sites since August 31, 2026, shows impressions in those features by page, country and device. It reports impressions, not clicks.
3. Bing Webmaster Tools. The AI Performance report, in public preview since February 10, 2026, counts citations across Copilot, Bing's AI summaries and some partner experiences, with the grounding queries and the cited pages. Bing notes that the figures say nothing about ranking or position within an answer.
4. Server logs. Crawls by OAI-SearchBot, PerplexityBot and Claude-SearchBot show which pages are in their indexes. Hits from ChatGPT-User, Perplexity-User and Claude-User show pages fetched live to answer a question, which is the closest thing to a citation log you can collect yourself. User agents are easy to fake, so check the source IPs against the ranges OpenAI, Perplexity and Anthropic publish.
5. A fixed prompt panel. Write 15 to 25 questions your buyers ask ("best open-source X for small teams", "X vs Y", "does X have a free plan"). Once a month, run them in each assistant, signed out where possible, and record whether the product is named, which URL is cited and whether the facts are right. Answers change from run to run, so read the trend over several months, not a single result.
Keep expectations in proportion. SE Ranking's study of 101,574 sites, published in June 2026, found that AI platforms sent 0.32% of all website traffic between January and April 2026, and that ChatGPT accounted for 74.78% of those AI referrals (SE Ranking). Judge the channel by sign-ups and installs per visit, not by visit counts alone.
What should a small software team do first?
In order of effort against likely effect:
- Check access. Confirm that robots.txt and your CDN allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot; that the Search Console generative AI control is set to include; that the site is verified in Bing Webmaster Tools; and that IndexNow is enabled if your platform supports it.
- Rewrite the first paragraph of the homepage, pricing page, docs landing page and README as a plain definition of the product.
- Date your facts. Add "as of" dates to pricing, "last updated" dates to docs and release dates to the changelog.
- Publish two or three honest comparison pages for the comparisons buyers actually make, plus one alternatives page if your category has a dominant incumbent.
- Add Organization and SoftwareApplication structured data that matches what the page shows.
- Make every third-party description match. GitHub, package registries, review sites and a short list of curated directories should carry the same one-line definition, current pricing and the same links.
- Set up measurement: the GA4 pattern, the Search Console and Bing reports, a monthly log count and the prompt panel.
- Update on every release. Change the facts pages and release notes the same day, and submit the changed URLs through IndexNow.
If a launch is coming up, fold steps 1 to 3 into your software launch checklist.
Skip the rest of what gets sold as AI optimization: llms.txt as a ranking lever, rewriting pages into fragments, paid mention schemes and separate pages written only for bots. Google's own guide lists most of these as things it ignores, and none of the other vendors document them as signals.
Frequently asked questions
If I block GPTBot, will my product disappear from ChatGPT?
No. GPTBot only tells OpenAI whether your pages may be used to train its models. Whether ChatGPT search can show and link your pages depends on OAI-SearchBot, which OpenAI documents as a separate setting. Disallow GPTBot and allow OAI-SearchBot if you want to appear in answers without contributing training data.
Can you pay to be cited in ChatGPT or Google's AI answers?
Neither OpenAI nor Google documents a paid route into the citations inside answers. ChatGPT does carry ads, which OpenAI checks with a separate crawler called OAI-AdsBot, but ads are a different product. Google's guide also warns that no third-party tool has access to its internal ranking or AI systems, so be wary of services that promise placement.
How quickly do changes show up in AI answers?
Access changes are quick: OpenAI says robots.txt updates reach its search systems in about a day, and Google applies a change to the Search Console generative AI control within a few days. Content changes wait for a recrawl. IndexNow speeds that up for Bing and other participating engines, and URL Inspection in Search Console lets you request a Google recrawl. Expect citations to shift over weeks rather than days.
Does an open-source project need to do anything different?
The priorities move toward the repository. Make the README's first paragraph a definition, keep the license, install steps and docs link current, publish dated release notes, and fill in the description and homepage fields on every package registry you publish to, so the project is described the same way wherever it is installed.
Does a product need a Wikipedia article to be recommended by AI assistants?
No. Wikipedia only accepts subjects with significant coverage in independent, reliable sources, and its conflict of interest rules discourage writing about your own company. Build that independent coverage first: reviews, articles and community discussion are also what assistants draw on when they compare products.
Is generative engine optimization different from SEO?
For Google, no: its guide says optimizing for generative AI search is still SEO. The differences come from the other assistants: separate crawlers to allow, Bing indexing that feeds Copilot and part of ChatGPT search, and new referrer domains to track in analytics.
Tools in this article
n8nFair-code workflow automation and AI agent platform with a visual editor, code steps and self-hosting
Facts link to their official sources. Product details can change after publication: report an error or read the editorial methodology.




