We run SongUp AI, an AI song generator, as a Next.js 14 App Router app on the edge runtime, deployed to Cloudflare Pages. It worked. It also quietly sabotaged our search visibility in three ways that none of the defaults warned us about.
Disclosure: I work on SongUp AI. Code below is simplified from our middleware.ts. Google indexed our preview deployments — and ranked one above the real domain
Every Cloudflare Pages deployment answers on ..pages.dev as well as on .pages.dev. Those URLs serve a byte-identical copy of the site. Google found them, indexed them, and for a while ranked a pages.dev URL above www.songupai.com for our own brand name.
We already had rel="canonical" pointing at the real domain. It didn't help: canonical is a hint, and Google can overrule it when it sees two identical sites.
What works is a directive: X-Robots-Tag: noindex on every response that isn't served from the real host.
The trap: don't also block those hosts in robots.txt. A disallowed URL is never recrawled, so Google never sees the noindex, and the already-indexed copies stay in the index indefinitely. Crawlable + noindex is what actually removes them. Locale from Accept-Language means crawlers only ever see English
Our UI is translated into 30 languages, and the locale is picked from the browser's Accept-Language header on the same URL. Great for users. Invisible to search engines: Googlebot doesn't send Accept-Language, so it only ever sees the English page, and there is nothing to put in hreflang.
The fix was a small set of real, crawlable localized URLs whose language comes from the path, not the header:
Then every page in the cluster declares the same hreflang set through the Metadata API:
app/sitemap.ts accepts the same object under alternates.languages, so the sitemap and the tags can't drift apart.
Two details that mattered: Don't write the locale cookie on the localized pages. A visitor who lands on /de from Google shouldn't have the rest of the site switch to German. Only localize what you can deliver. Our songs can be sung in eight languages, so we built pages only for languages the product actually supports. A /it page promising Italian songs we can't make would be worse than no page. (Ours are at songupai.com/de, /fr, /es and /pt if you want to see the result.) A Set-Cookie on every response makes every page uncacheable
Our middleware originally mirrored the detected locale into a NEXT_LOCALE cookie on every request. Any response with Set-Cookie is treated as private by caches, so the edge cache was effectively off for every page, including for crawlers.
The fix: English is the default everyone falls back to, so English responses get no cookie. Only set it when the locale differs from the default and from what the cookie already says; clear a stale one instead of rewriting it.
We applied the same idea to a GDPR-zone cookie: only visitors in the EU/EEA/UK/CH get it, so everyone else receives a cookie-free, cacheable response.
Google ignores IndexNow, but Bing uses it — and Bing's index feeds ChatGPT search and Microsoft Copilot. Setup is a text file and one POST:
A 202 means accepted with key validation pending. Run it after deploys that add or rewrite pages, not on every deploy.
Checklist [ ] X-Robots-Tag: noindex on every non-canonical host (.pages.dev, preview URLs) [ ] Those hosts are not** disallowed in robots.txt [ ] At least one crawlable URL per language you really support, with a matching hreflang cluster in pages and sitemap [ ] No Set-Cookie on default-locale responses [ ] IndexNow key file live and submissions after content deploys
If you've hit other Cloudflare Pages + Next.js SEO surprises, I'd like to hear them in the comments.