Skip to content

Journal

·6 min read

PerplexityBot, OAI-SearchBot, and the citation graph on your origin

Search-shaped crawlers fetch so an assistant can cite. Treat them as a graph of pages the models have actually read — not as “bot noise” to exclude and forget.

AI crawlersPerplexity

A cream printed page with one gold-highlighted sentence
A citation starts as a fetch. The graph is on your server.

GPTBot is the name everyone knows. The quieter ones — PerplexityBot, OAI-SearchBot, Google-Extended, ClaudeBot — are often the ones that precede a human click. Vector 5 maps each recognised crawler to a source family and keeps a public index you can fetch even if you never install the dashboard.

GET /api/crawler-index
GPTBot            → ChatGPT
OAI-SearchBot     → ChatGPT
ClaudeBot         → Claude
PerplexityBot     → Perplexity
Google-Extended   → Gemini

Once the middleware is on, you can see that Google-Extended read /docs while PerplexityBot preferred /blog. Pair that with human_via_ai from perplexity.ai the same week. That is a citation graph with two observable nodes: fetch, then person. Everything else is folklore.

Dark server corridor with amber lights
The index is public. The fetches are yours once the middleware is in the path.
A crawler you exclude “to clean the data” is often the only record that a model has read you.