Robots Meta Conflict
- 1 installs
- 73.4k repo stars
- Updated June 18, 2026
- thedaviddias/frontendchecklist
Finds pages blocked in robots.txt that also carry noindex, where the crawler can never read the tag, and makes them crawlable-but-noindex.
About
Detects the conflict where a robots.txt disallow prevents Google from ever reading a noindex tag, so the page stays indexed. A developer uses it when deindexed pages still appear in search.
- Blocked pages never get their noindex read
- To deindex, keep the URL crawlable but carry noindex
Robots Meta Conflict by the numbers
- 1 all-time installs (skills.sh)
- Ranked #1,710 of 1,879 Marketing & SEO skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/thedaviddias/frontendchecklist --skill robots-meta-conflictAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| repo stars | ★ 73.4k |
| Last updated | June 18, 2026 |
| Repository | thedaviddias/frontendchecklist ↗ |
What it does
Finds pages blocked in robots.txt that also carry noindex, where the crawler can never read the tag, and makes them crawlable-but-noindex.
Files
Robots Meta Conflict
Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt — Google cannot read the noindex directive it cannot fetch.
Quick Reference
- If a page is blocked in robots.txt, crawlers never fetch it and therefore never read its
noindexmeta tag - To properly deindex a page, use
<meta name="robots" content="noindex">without blocking it in robots.txt - If you block a URL in robots.txt, Google may still show it in results using anchor text from backlinks
- Audit pages that need to be deindexed by confirming they are crawlable but carry
noindex
Check
Cross-reference the site's robots.txt disallow paths against pages carrying <meta name="robots" content="noindex">. Flag any page where the URL matches a Disallow rule AND has a noindex tag — the noindex will never be seen by crawlers.
Fix
For pages that must be deindexed: remove the robots.txt Disallow rule for that URL and keep the noindex meta tag. For pages that must simply be uncrawled without appearing: keep the Disallow rule and remove the noindex tag (understanding Google may still show the URL).
Explain
Explain why a page blocked by robots.txt can still appear in Google search results, why the noindex meta tag inside a blocked page is never processed, and how to correctly deindex a page.
Code Review
Review metadata generation, rendered HTML, structured data, and response headers related to Robots Meta Conflict. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
---
For full implementation details, code examples, and framework-specific guidance, see references/rule.md.
Rule page: https://frontendchecklist.io/en/rules/seo/robots-meta-conflict
Robots Meta Conflict
Detects pages blocked by robots.txt that also carry noindex meta tags, creating a paradox where the directive is never read.
Priority: high · Difficulty: intermediate · Time: 10 min
--- A robots.txt Disallow rule and a noindex meta tag serve different purposes. Google's robots-meta documentation makes that distinction explicit, which is why this conflict often needs the same cleanup as broader indexability-conflicts.
Code Example
# robots.txt
User-agent: *
Disallow: /private/
# /private/page.html
<meta name="robots" content="noindex"> ← Google never reads thisWhen Googlebot sees Disallow: /private/, it stops crawling that path entirely. It never fetches /private/page.html, so it never processes the noindex tag.
Result: Google may still index the URL using signals from external backlinks, sitelinks, or previously cached content.
Why It Matters
Pages that appear in search results despite your intention to deindex them are usually blocked in robots.txt. Google cannot read the noindex directive it cannot fetch, so Google Search Console becomes the fastest way to confirm what Google actually saw.
Correct Patterns
#
✅ Deindex a page (make it crawlable but excluded from results)
# robots.txt — NO Disallow for this path
User-agent: *
Disallow: /admin/<!-- /thank-you.html -->
<head>
<meta name="robots" content="noindex, follow">
</head>Googlebot fetches the page, reads noindex, and removes it from search results.
✅ Block a staging environment completely
# robots.txt on staging.example.com
User-agent: *
Disallow: /This is acceptable on staging/preview environments where you simply do not want any content crawled at all.
❌ Conflicting setup (noindex never processed)
# robots.txt
User-agent: *
Disallow: /old-page/<!-- /old-page/index.html -->
<meta name="robots" content="noindex"> ← Never read by GooglebotQuick Reference
| Goal | robots.txt | meta robots |
|---|---|---|
| Allow crawling and indexing | No Disallow | index, follow (default) |
| Deindex page, allow link equity | No Disallow | noindex, follow |
| Block crawling (no deindex guarantee) | Disallow: /path/ | (irrelevant — not read) |
| Block private tool content | Disallow: /path/ | (not needed) |
Exceptions
- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.
Standards
- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google: Remove a page from Google before treating the rule as satisfied.
- Check the implementation against Google: robots meta tag documentation before treating the rule as satisfied.
Verification
Automated Checks
- Check Google Search Console → URL Inspection Tool for affected URLs
- Use the robots.txt tester to verify crawlability
Manual Checks
- Fetch the page as Google to confirm the meta robots tag is present and readable
Related skills
Seo AuditRun structured SEO audits on their SaaS site or content hub and receive a prioritized action plan.173k42.2kCopywritingGenerate, rewrite, or strengthen persuasive website and landing-page copy that converts visitors into users.164k42.2k
Viral Short FormQuickly generate high-retention hooks, scripts, and outlines for TikTok, Reels, YouTube Shorts, and carousels.132k67Viral HooksWrite and critique viral hooks for short-form video opening sequences.123k67Viral Captions And CtasOptimize social media captions and CTAs for viral short-form video reach and saves.123k67Viral Youtube ShortsWrite and diagnose YouTube Shorts for Shorts Feed and long-form funnel.123k67