Robots Meta

Name: Robots Meta
Author: thedaviddias

thedaviddias/frontendchecklist

1 installs
73.4k repo stars
Updated June 18, 2026
thedaviddias/frontendchecklist

Audits each page's robots meta directive to catch accidental noindex tags that silently remove key pages from search results.

About

Checks per-page robots meta directives so no landing page carries an unintended noindex that drops it from search. A developer uses it when pages disappear from results or checking staging vs production parity.

Use noindex only on pages that should not appear in search
Crawler-specific tags override the generic robots tag

Robots Meta by the numbers

1 all-time installs (skills.sh)
Ranked #1,710 of 1,879 Marketing & SEO skills by installs in the Skillselion catalog
Data as of Aug 5, 2026 (Skillselion catalog sync)

npx skills add https://github.com/thedaviddias/frontendchecklist --skill robots-meta

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/thedaviddias/frontendchecklist/robots-meta.svg)](https://skillselion.com/skills/thedaviddias/frontendchecklist/robots-meta)

Installs	1
repo stars	★ 73.4k
Last updated	June 18, 2026
Repository	thedaviddias/frontendchecklist ↗

What it does

Audits each page's robots meta directive to catch accidental noindex tags that silently remove key pages from search results.

Files

SKILL.mdMarkdownGitHub ↗

Set robots meta directives correctly

An accidental noindex on a key landing page silently removes it from search results; audit every page's robots directive to avoid invisible ranking losses.

Quick Reference

Place <meta name="robots" content="..."> in <head> to control indexing per-page
Use noindex only on pages that should not appear in search results
Avoid noindex, nofollow on pages you want crawled for PageRank flow
Crawler-specific tags (googlebot, bingbot) override the generic robots tag for that crawler

Check

Inspect <head> for <meta name="robots" content="...">. Verify the content value is intentional (e.g., production pages should not have noindex). Check for conflicting googlebot-specific tags.

Fix

Remove or update the noindex directive from pages that should appear in search results. For pages that must be blocked, confirm noindex is intentional. Remove duplicate or conflicting robots meta tags.

Explain

Explain the difference between robots.txt and meta robots, when each should be used, and why noindex combined with follow still allows link equity to flow.

Code Review

Review metadata generation, rendered HTML, structured data, and response headers related to Set robots meta directives correctly. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.

---

For full implementation details, code examples, and framework-specific guidance, see references/rule.md.

Rule page: https://frontendchecklist.io/en/rules/seo/robots-meta

Set robots meta directives correctly

Checks robots meta tag for valid indexing directives in the page head.

Priority: high · Difficulty: beginner · Time: 10 min

--- The robots meta tag gives per-page control over indexing and link following, unlike robots.txt which operates at the URL path level. Google's robots-meta guide and broader indexability checks should agree on the same intent for every page.

Code Example

<head>
  <!-- Default: allow indexing and link following -->
  <meta name="robots" content="index, follow">

  <!-- Prevent this page from appearing in search results -->
  <meta name="robots" content="noindex, follow">

  <!-- Index but don't pass link equity through outbound links -->
  <meta name="robots" content="index, nofollow">

  <!-- Block entirely: no index, no link following -->
  <meta name="robots" content="noindex, nofollow">
</head>

Why It Matters

An accidental noindex on a key landing page silently removes it from search results. Auditing those directives in Google Search Console is often the quickest way to confirm whether a disappearance is technical rather than content-related.

Supported Directives

Directive	Meaning
`index`	Page may appear in search results (default)
`noindex`	Page must not appear in search results
`follow`	Crawlers may follow links (default)
`nofollow`	Crawlers must not follow links on this page
`noarchive`	Do not show a cached link in results
`nosnippet`	Do not show a text snippet or video preview
`max-snippet:-1`	No limit on snippet length
`max-image-preview:large`	Allow large image previews
`noimageindex`	Do not index images on this page

Crawler-Specific Tags

<!-- Only applies to Googlebot -->
<meta name="googlebot" content="noindex">

<!-- Only applies to Bingbot -->
<meta name="bingbot" content="noindex">

Crawler-specific tags take precedence over the generic robots tag for that crawler.

✅ Correct Usage

<!-- Public article page -->
<meta name="robots" content="index, follow">

<!-- Thank-you page after purchase -->
<meta name="robots" content="noindex, nofollow">

<!-- Paginated page — index but limit snippet -->
<meta name="robots" content="index, follow, max-snippet:150">

❌ Common Mistakes

<!-- Accidentally left on production from staging -->
<meta name="robots" content="noindex">

<!-- Conflicts: two robots tags on the same page -->
<meta name="robots" content="index, follow">
<meta name="robots" content="noindex">

<!-- Blocking a page in robots.txt AND adding noindex is redundant -->
<!-- robots.txt block prevents the noindex from even being seen -->

robots.txt vs. Meta Robots

robots.txt blocks access — the page is never fetched
meta robots noindex blocks indexing — the page is fetched but excluded from results
For pages you want indexed: neither block nor noindex
For pages you want deindexed: use noindex (not robots.txt, which prevents Google from seeing the directive)

Exceptions

Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.

Standards

Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
Check the implementation against Google: robots meta tag, data-nosnippet, and X-Robots-Tag before treating the rule as satisfied.
Check the implementation against Google: Valid indexing and serving rules before treating the rule as satisfied.

Verification

Automated Checks

Inspect rendered HTML and HTTP headers to confirm the expected metadata or crawlability signal is present.
Test the affected URL with Google Search Console or equivalent tooling where relevant.
Re-crawl a representative page set after deployment.