Skip to main content
Serving different content to a crawler than you serve to human visitors is called cloaking, and Google penalizes it. Bluesky’s Agent Edge Network (AEN) is built around a firewall that makes a hard distinction between classic search-index bots — which must always see your real page — and non-search answer-engine agents, which can safely receive a more machine-readable form of the same content.

​
What cloaking is and why it matters

Search engines like Google send crawlers to build their index. If those crawlers see content that differs from what users see, Google considers it an attempt to manipulate rankings and can demote or deindex the affected pages. The risk is real even when the intent is benign: serving any content variant to Googlebot, including a “nicer” version, crosses that line.
Serving a different page to Googlebot — even a better one — is a Google Search policy violation. The cloaking firewall exists to ensure that never happens, regardless of how you configure the rest of the AEN.

​
How Bluesky stays on the right side of the line

Bluesky’s approach rests on two guarantees that work together.

​
Guarantee 1: Search-index bots are never bot-branched

The Bluesky bot registry marks every classic search-index crawler with isSearchIndex: true. The middleware checks this flag before it evaluates verification, purpose, or serve mode, and if the flag is set it immediately returns NextResponse.next() — the same code path a normal visitor takes.
No artifact is fetched for this branch. The request reaches your Next.js application exactly as a normal visitor’s would. The bots that carry isSearchIndex: true are:
BotOperator
GooglebotGoogle
BingbotMicrosoft
ApplebotApple
This list lives in the Bluesky-managed bot registry (the botset), not in your local middleware file. The middleware fetches the botset from the control plane and caches it. When a vendor publishes a new crawler variant, Bluesky updates the registry and your edge picks it up on the next refresh — no redeployment required.

​
Why Google’s AI products are covered by this rule

You might wonder: if Google’s AI Overviews use different content than classic search, why not serve a special version to “Google’s AI crawler”? The answer is that Google AI Overviews run off the same Googlebot and the same search index as Google Search. There is no separate crawler for AI Overviews. If you served a different page to Googlebot, that different page is what both AI Overviews and Search would index and rank you on — there is no way to target one without cloaking the other. This is why the rule is unconditional: a search-index bot always sees your real page, full stop.

​
Guarantee 2: Content parity — same substance, better form

The AEN only varies the form of your content, never the substance. The optimized artifact that verified non-search agents receive is:
  • Answer-first markdown with embedded Article JSON-LD (plus FAQPage JSON-LD when your page contains question-and-answer content)
  • Derived entirely from your existing page — the generator runs a strict-parity grounding guardrail and will not add claims, facts, or assertions that are not already present in the source
  • Generated offline and cached, so there is no model call on the request path

What changes

Form. The content is restructured as answer-first prose with structured data that helps answer engines extract specific facts without parsing HTML.

What stays the same

Substance. Every claim in the artifact exists on your source page. The generator refuses to emit an artifact that introduces new assertions. If your page changes, you regenerate the artifact.
This means that even if a search engine somehow crawled the artifact URL, the information it would index is the same information on your page — just more cleanly structured.

​
Which bots are served the optimized artifact

Only verified non-search answer-engine agents are eligible for bot-branching — crawlers operated by products that answer user questions directly rather than building a ranked index.
BotOperatorPurpose class
OAI-SearchBotOpenAIRETRIEVAL
ChatGPT-UserOpenAIUSER_TRIGGERED
Claude-SearchBotAnthropicRETRIEVAL
PerplexityBotPerplexityRETRIEVAL
None of these carry isSearchIndex, so they pass the firewall check and enter the detection-and-serve flow.
In the drop-in Next.js middleware, the ENABLED set controls which purpose classes are eligible. The default is new Set(["USER_TRIGGERED", "RETRIEVAL"]). Narrow it if you only want to serve one class of agent. See serve modes for how shadow, canary, and default gate whether an eligible agent is actually served.

​
The two-tier bot model

The drop-in middleware verifies an agent by matching its source IP against the vendor’s published CIDR ranges (IP_VERIFIED). A bot recognized by user agent alone is treated as unverified (UA_ONLY) and always gets the normal page. The firewall is not a configuration option — the isSearchIndex check is hardcoded and runs first. You cannot disable it by changing SERVE_MODE or ENABLED. Classic search-index bots always receive your normal page.

Serve modes

off, shadow, canary, and default — how the AEN ramps from log-only to full serving.

Agent analytics

See which agents hit your site, how they verified, and served-vs-cited breakdowns.