Skip to main content
When something isn’t working the way you expect, most issues trace back to one of a handful of root causes: the middleware isn’t deployed, the edge token is missing, a bot didn’t meet the verification threshold, or the serve mode is still in shadow. Work through the section below that matches what you’re seeing.
If you’ve deployed the middleware and triggered an AI crawler but Agent Analytics is still empty, check each of the following in order.
1

Confirm the middleware is deployed

In Vercel, open your project and check the Functions tab. If middleware isn’t listed after a deploy, your middleware.ts file may not be at the root of your Next.js project, or the deployment that included it hasn’t finished. Re-deploy and check again.
2

Confirm AEN_EDGE_TOKEN is set in Vercel

The token must be set as a Vercel environment variable — not just in a local .env file. Go to Vercel → your project → Settings → Environment Variables and confirm AEN_EDGE_TOKEN is present and non-empty. The middleware reads it as process.env.AEN_EDGE_TOKEN; if it’s blank, the middleware calls NextResponse.next() on the first line and returns before any detection runs.
3

Trigger a real answer-engine crawler

The middleware only records a hit when a request’s user-agent matches a known answer-engine bot token from the botset. Browsing your own site won’t do it. To generate a hit, open ChatGPT or Perplexity with browsing enabled and ask a question that makes it fetch a URL on your domain. Its crawler (for example ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-SearchBot) sends a request the middleware can match.
4

Note which crawlers are not recorded

Two cases produce no telemetry at all: a request whose user-agent matches no known bot token, and a classic search-index bot. Googlebot, Bingbot, and Applebot pass straight through the cloaking firewall and are never recorded — so if the only crawler that reached your page was Googlebot, Agent Analytics stays empty by design.
5

Verification level does not gate recording

A common misconception: a matched answer-engine bot is recorded whether or not its IP verified. A request whose user-agent matches but whose source IP is not inside the vendor’s published CIDR ranges is recorded with trust level UA_ONLY; a request from a published vendor IP is recorded as IP_VERIFIED. Both show up in the By verification level breakdown. Verification only decides whether the optimized variant is served — not whether the hit is counted.
6

Check your middleware matcher

The default matcher covers all public pages but excludes _next/, api/, and favicon.ico. If the URL you tested falls outside the matcher’s scope, the middleware never runs for it.
If you’ve customised this matcher, make sure the URL you tested is still covered.
Each recorded hit is reported via a fire-and-forget call to /api/v1/edge/telemetry — it never blocks your page response. If network conditions caused that call to fail silently, the hit won’t appear. Re-testing with a fresh crawler request is the fastest way to confirm.
The middleware can detect and record a crawler and still serve your normal page. There are several legitimate reasons — work through each one.
1

Confirm SERVE_MODE is set to default

The drop-in middleware.ts ships with SERVE_MODE set to "shadow" by design — a safe rollout mode that records what it would serve but always delivers your normal page. To actually serve the optimized variant, change the constant and re-deploy:
In "shadow", no variant is ever served, regardless of verification level.
2

Confirm the bot was IP_VERIFIED

Only a verified request is eligible for the variant. In this drop-in, verification is by published vendor IP ranges (CIDR), so the eligible trust level is IP_VERIFIED. A UA_ONLY request — user-agent matched but source IP not in a published range — always receives your normal page, even in "default" mode. Check the By verification level breakdown in Agent Analytics.
3

Verify the bot's purpose class

Even a verified bot is only served the variant if its purpose is USER_TRIGGERED or RETRIEVAL (the middleware’s enabled set). Bots with other purposes — for example training crawlers like GPTBot or ClaudeBot — pass through to your normal page. Check the By agent purpose breakdown for the hit in question.
4

Check whether a fresh artifact exists for the URL

The variant is only served if a pre-generated artifact exists for that exact URL and its status is fresh. The middleware requests /api/v1/edge/artifact per hit; if the control plane has no fresh artifact it returns a 404 and the middleware falls through to your normal page. Find the URL in the Optimized pages list in Agent Analytics and check its status. If it’s missing or not fresh, use Regenerate to build one.
All four conditions must hold at once: SERVE_MODE = "default", the bot is IP_VERIFIED, its purpose is USER_TRIGGERED or RETRIEVAL, and a fresh artifact exists for the URL. If any one fails, the middleware serves your normal page.
If requests pass through without any detection, telemetry, or serving, the middleware itself likely isn’t executing.
1

Verify the file name and location

Next.js only recognises the middleware when it is named exactly middleware.ts (or middleware.js) and placed at the root of your project — the same directory as package.json. Placing it inside src/, app/, pages/, or any other subdirectory makes Next.js ignore it silently.
2

Confirm you're deployed on Vercel Edge

The drop-in imports next/server and runs on the Vercel Edge runtime. If you’ve customised your project’s runtime settings or are self-hosting outside Vercel, confirm the middleware appears under the project’s Functions tab after deploying.
3

Check that AEN_EDGE_TOKEN is non-empty

The first line of the middleware checks the edge token. If process.env.AEN_EDGE_TOKEN is empty or undefined, the middleware calls NextResponse.next() and returns immediately — no detection, no telemetry, no serving. Confirm the variable is set in Vercel and that you redeployed after adding it.
If you’re testing locally with next dev, add AEN_EDGE_TOKEN to your .env.local. Production deployments read it from Vercel’s environment variable store instead.
This is the most common concern for anyone new to serving alternate content to crawlers. The Agent Edge Network is designed to stay on the safe side of the cloaking line.
1

Understand the cloaking firewall

Before any serve decision, the middleware checks whether the matched bot has isSearchIndex set. If it does, it calls NextResponse.next() and returns your normal page — no variant, no branching, and no telemetry. Googlebot, Bingbot, and Applebot always receive your normal page, without exception.
2

Understand why this matters for AI Overviews

Google’s AI Overviews run on the same Googlebot crawl and search index as Google Search. Branching Googlebot would cloak Search itself — a clear policy violation. By always passing search-index bots through, the network keeps your Google rankings and AI Overview eligibility unaffected.
3

Understand content parity

For the non-search answer-engine agents that are served (retrieval and user-triggered bots such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, and ChatGPT-User), the served variant carries the same substance as your page — restructured into answer-first markdown that machines parse more reliably. (The offline generator also produces schema.org JSON-LD as part of the stored artifact, though this drop-in serves the markdown form.) The generator restructures your existing page under a strict-parity grounding guardrail: it never publishes anything your source page didn’t already say.
Serving a more machine-readable form of the same content to non-search answer engines is not cloaking — it is the same idea as a print stylesheet or an RSS feed. The substance is identical; only the format differs.
Artifacts are generated offline from a snapshot of your page content, then cached. If you’ve edited a page since its artifact was generated, the served variant can lag the live page.
1

Use the Regenerate action

In Agent Analytics, find the page in the Optimized pages list and click Regenerate. The engine rebuilds the artifact from the current state of your page.
2

How stale artifacts behave

The artifact endpoint only returns artifacts whose status is fresh — a non-fresh artifact returns a 404 and the middleware serves your normal page rather than stale content. The out-of-date case to watch for is an artifact still marked fresh that was generated before your edit: it keeps serving the older content until you Regenerate.
3

No edge redeploy needed

This drop-in fetches the artifact from the control plane on every qualifying request — it does not cache artifacts at the edge. A freshly regenerated artifact is therefore available to the very next qualifying crawler request, with no redeploy or cache purge on your side.
4

Note the botset cache window

The botset (the list of known AI agents and their IP ranges) is cached in the middleware for up to one hour to avoid a control-plane call on every request. This is independent of artifacts — artifacts are fetched per request every time.
Still stuck? Review Serve modes to confirm your rollout stage, or walk back through the Quickstart to re-check token and deployment steps.