> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trybluesky.com/llms.txt
> Use this file to discover all available pages before exploring further.
# The cloaking firewall
> How the Agent Edge Network keeps classic search bots on your real page while serving optimized content only to verified answer-engine agents.
Serving different content to a crawler than you serve to human visitors is called *cloaking*, and Google penalizes it. Bluesky's Agent Edge Network (AEN) is built around a firewall that makes a hard distinction between classic search-index bots — which must always see your real page — and non-search answer-engine agents, which can safely receive a more machine-readable form of the same content.
## What cloaking is and why it matters
Search engines like Google send crawlers to build their index. If those crawlers see content that differs from what users see, Google considers it an attempt to manipulate rankings and can demote or deindex the affected pages. The risk is real even when the intent is benign: serving *any* content variant to Googlebot, including a "nicer" version, crosses that line.
**Serving a different page to Googlebot — even a better one — is a Google Search policy violation.** The cloaking firewall exists to ensure that never happens, regardless of how you configure the rest of the AEN.
## How Bluesky stays on the right side of the line
Bluesky's approach rests on two guarantees that work together.
### Guarantee 1: Search-index bots are never bot-branched
The Bluesky bot registry marks every classic search-index crawler with `isSearchIndex: true`. The middleware checks this flag **before** it evaluates verification, purpose, or serve mode, and if the flag is set it immediately returns `NextResponse.next()` — the same code path a normal visitor takes.
```ts theme={null}
// CLOAKING FIREWALL: classic search bots (Googlebot/Bingbot/Applebot) → always the human page.
if (bot.isSearchIndex) return NextResponse.next();
```
No artifact is fetched for this branch. The request reaches your Next.js application exactly as a normal visitor's would.
The bots that carry `isSearchIndex: true` are:
| Bot | Operator |
| --------- | --------- |
| Googlebot | Google |
| Bingbot | Microsoft |
| Applebot | Apple |
This list lives in the Bluesky-managed bot registry (the botset), not in your local middleware file. The middleware fetches the botset from the control plane and caches it. When a vendor publishes a new crawler variant, Bluesky updates the registry and your edge picks it up on the next refresh — no redeployment required.
### Why Google's AI products are covered by this rule
You might wonder: if Google's AI Overviews use different content than classic search, why not serve a special version to "Google's AI crawler"?
The answer is that **Google AI Overviews run off the same Googlebot and the same search index as Google Search**. There is no separate crawler for AI Overviews. If you served a different page to Googlebot, that different page is what both AI Overviews and Search would index and rank you on — there is no way to target one without cloaking the other.
This is why the rule is unconditional: a search-index bot always sees your real page, full stop.
### Guarantee 2: Content parity — same substance, better form
The AEN only varies the *form* of your content, never the *substance*. The optimized artifact that verified non-search agents receive is:
* Answer-first markdown with embedded `Article` JSON-LD (plus `FAQPage` JSON-LD when your page contains question-and-answer content)
* Derived entirely from your existing page — the generator runs a strict-parity grounding guardrail and will not add claims, facts, or assertions that are not already present in the source
* Generated offline and cached, so there is no model call on the request path
**Form.** The content is restructured as answer-first prose with structured data that helps answer engines extract specific facts without parsing HTML.
**Substance.** Every claim in the artifact exists on your source page. The generator refuses to emit an artifact that introduces new assertions. If your page changes, you regenerate the artifact.
This means that even if a search engine somehow crawled the artifact URL, the information it would index is the same information on your page — just more cleanly structured.
## Which bots are served the optimized artifact
Only verified non-search answer-engine agents are eligible for bot-branching — crawlers operated by products that answer user questions directly rather than building a ranked index.
| Bot | Operator | Purpose class |
| ---------------- | ---------- | ---------------- |
| OAI-SearchBot | OpenAI | `RETRIEVAL` |
| ChatGPT-User | OpenAI | `USER_TRIGGERED` |
| Claude-SearchBot | Anthropic | `RETRIEVAL` |
| PerplexityBot | Perplexity | `RETRIEVAL` |
None of these carry `isSearchIndex`, so they pass the firewall check and enter the detection-and-serve flow.
In the drop-in Next.js middleware, the `ENABLED` set controls which purpose classes are eligible. The default is `new Set(["USER_TRIGGERED", "RETRIEVAL"])`. Narrow it if you only want to serve one class of agent. See [serve modes](/guides/serve-modes) for how `shadow`, `canary`, and `default` gate whether an eligible agent is actually served.
## The two-tier bot model
```
Incoming matched bot request
│
▼
isSearchIndex?
┌── yes ──────────────────────────────▶ NextResponse.next()
│ (Googlebot, Bingbot, Applebot)
└── no
│
▼
IP-verified + purpose in ENABLED?
┌── yes ── serve mode allows ─────────▶ Serve optimized artifact
│ (OAI-SearchBot, Claude-SearchBot,
│ PerplexityBot, ChatGPT-User…)
└── no / shadow / out-of-canary
│
▼
NextResponse.next()
```
The drop-in middleware verifies an agent by matching its source IP against the vendor's published CIDR ranges (`IP_VERIFIED`). A bot recognized by user agent alone is treated as unverified (`UA_ONLY`) and always gets the normal page.
The firewall is not a configuration option — the `isSearchIndex` check is hardcoded and runs first. You cannot disable it by changing `SERVE_MODE` or `ENABLED`. Classic search-index bots always receive your normal page.
## Related
off, shadow, canary, and default — how the AEN ramps from log-only to full serving.
See which agents hit your site, how they verified, and served-vs-cited breakdowns.