The noindex that shipped with a release

Three times this year, in three different companies, the same thing. A team ships a release. Templates get rebuilt, a framework gets upgraded, or a staging environment gets promoted. Somewhere in the change, a noindex tag that belonged to staging comes along for the ride. The site works. Users see nothing wrong. Search engines quietly drop the affected pages over the following weeks.

I am writing this down because every one of those teams had good engineers and a decent QA process, and none of them caught it. The fault is not a skills problem. It is a visibility problem: a robots tag is invisible to everyone except a crawler.

How it happens

Staging environments are noindexed for good reasons. The tag is usually set in one of three places: an environment variable that the layout reads, a meta tag hard-coded into a template while someone was testing, or a header set at the CDN for the staging hostname. Each of those has a way of surviving the trip to production.

• The environment variable is copied to the new production config during a platform change, because the config was copied wholesale.
• The hard-coded tag is in a template that nobody expected to ship, and then a refactor moves that template into the main layout.
• The CDN rule matches on a path rather than a hostname, and the path exists in production too.

In one case the tag only applied to a category template. The home page and the articles were fine, which is why the traffic chart did not fall off a cliff. It sagged. Someone blamed seasonality. Two months later the category pages were gone from the index, and with them most of the non-branded revenue.

Why nobody sees it

Nothing in the normal release process reads the head of the document the way a crawler does. Visual QA passes. Automated tests pass, because they test behaviour, not metadata. Analytics keeps counting the visitors who are already there. Search Console will tell you eventually, under ‘Excluded by noindex tag’, but nobody opens that report on release day.

The delay is what makes it expensive. By the time the drop is visible in revenue, several more releases have shipped, and the person investigating has to work backwards through all of them.

The four-minute check

We now put five lines in every client’s release checklist, and we run the same five ourselves before we deploy anything. They take about four minutes by hand and a few seconds when scripted.

1. 01

   Fetch one page from each money template on the live hostname

   Home, a category, a product, an article, a landing page. Use curl or the browser’s view-source, not the rendered DOM.

2. 02

   Check for noindex

   In the meta robots tag and in the X-Robots-Tag response header. Both. The header is the one people forget.

3. 03

   Check the canonical

   It should point at the production hostname, not staging, and not at a different page.

4. 04

   Check robots.txt

   Diff it against the last known good version. A staging robots.txt that disallows everything is a classic.

5. 05

   Confirm the main content is in the server HTML

   If the product grid or the article body only exists after a script runs, the page is a shell to a crawler.

The bottom line

• The fault is common, silent and expensive, and it is carried by releases.
• Read the head of one page per template after every deployment: robots meta, X-Robots-Tag, canonical, robots.txt, server-rendered content.
• Turn the check into a build test if you can. Checklists get skipped; tests do not.

How this guide was put together

• An anonymised pattern from three engagements in 2026. Details that could identify a client have been removed; the mechanism is described as observed.

Sources

1. Google Search Central: Block search indexing with noindex
2. Google Search Central: Robots meta tag and X-Robots-Tag specifications
3. Lucidens: The Technical SEO Audit Checklist for 2026