> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trybluesky.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Crawler Detection and Verification Levels

> How Bluesky identifies AI crawlers by published IP ranges (and, in the core, cryptographic signatures), and what each verification tier means for what gets served.

Not all requests claiming to be from an AI crawler actually are. User-agent strings are trivially spoofable — anyone can send `User-Agent: GPTBot` from their laptop. The Bluesky edge middleware treats a matching user-agent as a first filter only, then verifies the request's **source IP** against the vendor's published CIDR ranges. Only requests that pass a real verification check are ever served the optimized artifact.

## Why user-agent alone isn't enough

A user-agent string is a plain-text header. There is nothing stopping a bad actor — or a curious developer — from sending a request with `User-Agent: PerplexityBot` from any IP address. (Cloudflare delisted PerplexityBot for exactly this kind of stealth crawling — the canonical proof that a UA token alone is worthless.) If Bluesky served the optimized variant to every request that looked like a known bot, you'd be handing your AEO content to anyone who asked.

The middleware therefore treats UA matching as a necessary first step, not a sufficient one:

1. The UA is scanned for a known bot token — if nothing matches, the request is ignored entirely (passthrough, no telemetry).
2. If a bot token matches, the request's source IP is checked against that bot's published CIDR ranges.
3. Only requests that verify at step 2 are eligible for the variant.

## Verification tiers

The AEN core arranges verification into ascending-trust tiers. The four that bear on serving are below; only `SIGNED` and `IP_VERIFIED` are treated as **verified** and are ever eligible to receive the variant.

### `UA_ONLY`

The user-agent matched a known bot pattern, but the source IP is not inside any of the vendor's published CIDR ranges. This is the lowest tier and is treated as unverified.

**Result:** The request always receives your normal page. `UA_ONLY` hits are still logged so you can see them in Agent Analytics, but they are never served the optimized artifact.

### `IP_VERIFIED`

The user-agent matched *and* the source IP falls within one of the vendor's published CIDR ranges. This is the tier the shipped drop-in middleware produces for verified traffic. The drop-in itself is test-grade — it fetches config and artifacts from the control plane on every request (the productized edge caches those calls), so treat it as a working starting point rather than a tuned production deployment.

**Result:** Eligible to receive the optimized variant, subject to serve mode and purpose class.

### `DNS_VERIFIED`

Forward-confirmed reverse DNS (rDNS → forward DNS) on the source IP. This tier is defined in the core but is resolved **engine-side / asynchronously** — it is deliberately kept off the edge hot path because it adds latency. The drop-in middleware does not perform rDNS, and the edge serve rule does not treat `DNS_VERIFIED` as serve-eligible.

### `SIGNED`

The request carries a valid [Web Bot Auth](https://www.rfc-editor.org/rfc/rfc9421.html) signature (RFC 9421 HTTP Message Signatures, Ed25519 profile), verified against the vendor's pinned public keys. This is the strongest tier — the signature is cryptographically bound to the request and is the only method that survives shared-cloud or residential-proxy IPs.

**Result:** Eligible to receive the optimized variant, subject to serve mode and purpose class.

<Note>
  Web Bot Auth (`SIGNED`) verification lives in the AEN core (`webbotauth.ts`) and runs in adapters that wire it in. The shipped **drop-in Next.js middleware is IP-only** — it verifies by published CIDR ranges and emits either `IP_VERIFIED` or `UA_ONLY`. Signature-based `SIGNED` and reverse-DNS `DNS_VERIFIED` are core/engine capabilities not exercised by the IP-only drop-in.
</Note>

## How IP verification works (drop-in middleware)

The middleware fetches the current bot configuration (including CIDR ranges) from Bluesky's control plane via `GET /api/v1/edge/botset` and caches it for one hour:

```ts theme={null}
async function getBots(): Promise<Bot[]> {
  if (botsetCache && Date.now() - botsetCache.at < 3_600_000) return botsetCache.bots;
  try {
    const r = await fetch(`${AEN_BASE}/api/v1/edge/botset`, { headers: auth() });
    const j = await r.json();
    const bots: Bot[] = j?.botset?.bots ?? [];
    botsetCache = { at: Date.now(), bots };
    return bots;
  } catch {
    return botsetCache?.bots ?? [];
  }
}
```

For each request, the source IP is extracted from `x-forwarded-for` (or `x-real-ip`) and tested against each CIDR in the matched bot's range list:

```ts theme={null}
function ipInCidrs(ip: string, cidrs: string[] = []): boolean {
  for (const c of cidrs) {
    try {
      const [net, bitsRaw] = c.split("/");
      const bits = Number(bitsRaw);
      if (ip.includes(":") || net.includes(":")) continue; // IPv4 only in this drop-in
      const a = ipToInt(ip), b = ipToInt(net);
      if (a === null || b === null) continue;
      const mask = bits === 0 ? 0 : (~0 << (32 - bits)) >>> 0;
      if ((a & mask) === (b & mask)) return true;
    } catch { /* skip */ }
  }
  return false;
}
```

If the IP matches any of the ranges, the request is classified as `IP_VERIFIED`. CIDR ranges refresh from the control plane on the next cache cycle, so your deployed middleware picks up vendor changes without a redeploy.

<Note>
  The drop-in matcher is **IPv4-only** (it skips any IPv6 address or CIDR). The portable AEN core (`botset.ts`) implements full IPv4 **and** IPv6 bit-prefix matching, and — like the drop-in — fails closed on any malformed or ambiguous input.
</Note>

## How bot matching works

The middleware scans the incoming user-agent for a known token using a word-boundary check. When multiple bot definitions match, the one with the **longest token wins** — this prevents a short token from incorrectly matching a more specific agent (e.g. a hypothetical `Claude` never masks `Claude-SearchBot`):

```ts theme={null}
function matchBot(ua: string, bots: Bot[]): Bot | null {
  const low = ua.toLowerCase();
  let best: Bot | null = null;
  for (const b of bots) {
    const t = b.uaToken.toLowerCase();
    if (new RegExp(`(?:^|[^a-z0-9])${escapeRe(t)}(?:[^a-z0-9]|$)`, "i").test(low)) {
      if (!best || b.uaToken.length > best.uaToken.length) best = b;
    }
  }
  return best;
}
```

Matching is on the **token only**, never the version suffix — so `GPTBot/1.3` still matches `GPTBot`. The list of known tokens and their metadata is maintained by Bluesky and delivered via `GET /api/v1/edge/botset`; you don't manage bot definitions yourself.

## Purpose classes

Every matched bot carries a **purpose class** describing why it is fetching your page:

| Purpose          | Served? | Description                                                                                                         |
| ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------- |
| `USER_TRIGGERED` | Yes     | A user asked an AI assistant to browse your page in real time (e.g. `ChatGPT-User`, `Perplexity-User`)              |
| `RETRIEVAL`      | Yes     | An answer engine fetching to build or refresh its index (e.g. `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) |
| `TRAINING`       | No      | Corpus / training crawlers (e.g. `GPTBot`, `ClaudeBot`, `CCBot`)                                                    |
| `SEARCH_INDEX`   | No      | Classic search crawlers — always passthrough (see cloaking firewall below)                                          |
| `UNKNOWN`        | No      | No token matched                                                                                                    |

The drop-in middleware enables only the two answer-engine purposes:

```ts theme={null}
const ENABLED = new Set(["USER_TRIGGERED", "RETRIEVAL"]);
```

A request must be both **verified** (`IP_VERIFIED` or `SIGNED`) **and** in an enabled purpose class to be eligible for the variant. Training crawlers and unknown agents are never served, even when their IP verifies.

## The cloaking firewall

Before any verification or serve-mode logic runs, the middleware checks whether the matched bot is a classic search crawler:

```ts theme={null}
// CLOAKING FIREWALL: classic search bots (Googlebot/Bingbot/Applebot) → always the human page.
if (bot.isSearchIndex) return NextResponse.next();
```

`Googlebot`, `Bingbot`, and `Applebot` are the only entries flagged `isSearchIndex: true`. They are passed through unconditionally — no variant is considered and no telemetry is logged. This is a hard safeguard that cannot be overridden by serve mode or the enabled-purpose set.

<Warning>
  AI Overviews run on Googlebot and the same search index. Branching the response for those crawlers would cloak Search itself — which Google penalizes. The firewall makes that impossible, even if a configuration change accidentally enabled a search-index bot.
</Warning>

The answer-engine agents Bluesky *does* serve — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `ChatGPT-User`, and similar — are distinct from the search-index crawlers. They power AI-generated answers, not organic search results, so varying the response format for them is not cloaking.

## Content parity

Even for fully verified, eligible requests, the optimized artifact holds **strict content parity** with your original page. The Bluesky generator produces answer-first markdown and `Article` / `FAQPage` JSON-LD by extracting and restructuring your existing page — it never fabricates claims that aren't in the source. Artifacts are generated offline and cached, so there is no model call on the request path. Same substance, different form — an additional guarantee against any reading of cloaking.

## Verification in Agent Analytics

Every logged hit records the verification tier alongside the bot name, path, purpose class, and served status, so you can filter to verified traffic or review `UA_ONLY` hits to spot crawlers operating from unregistered IPs.

<Tip>
  If a known AI crawler consistently lands as `UA_ONLY`, it may be operating from an IP range not yet in Bluesky's botset. Reach out and we can add it.
</Tip>

## Related

<CardGroup cols={2}>
  <Card title="Serve modes" href="/guides/serve-modes">
    off / shadow / canary / default — how the verified-and-eligible decision turns into an actual served response.
  </Card>

  <Card title="Quickstart" href="/quickstart">
    Provision a domain, get your edge token, and drop the middleware into your Next.js app.
  </Card>
</CardGroup>