WordPress robots.txt: What to Block, What Not To, and the Virtual File Problem

WordPress generates robots.txt dynamically. There is no file on disk unless you created one, which is why searching the server for it finds nothing and why editing it means either creating a real file or hooking a filter. The more consequential point is that most of what people put in robots.txt makes their SEO worse rather than better.

The default WordPress robots.txt is short and close to correct. The damage comes from copying an elaborate one from a forum post written in 2011, when crawlers worked differently and blocking assets did not break rendering.

The virtual file
What not to block
Why robots.txt does not remove pages from the index
What is actually worth blocking
When crawl budget matters
Checking it works
How this fits the rest of the stack
FAQ

The virtual file

Visit /robots.txt on a stock WordPress site and you get something like:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

That is generated at request time by WordPress, not read from a file. Two ways to change it.

Create a real file at the site root. WordPress detects it and stops generating the virtual one. Simple, and it means the file is separate from your code and easy to forget about.

Filter it, which keeps it in your theme or plugin and therefore in version control:

add_filter( 'robots_txt', function( $output ) {
    $output .= "Disallow: /?s=\n";
    return $output;
}, 10, 1 );

One thing that surprises people: if the site’s Reading settings have search engine visibility discouraged, WordPress serves a robots.txt disallowing everything and the filter output is irrelevant. That setting is the single most common cause of a site not being indexed at all, and it is usually left on from development.

What not to block

The elaborate robots.txt files circulating online do real damage. Three things people commonly block that they should not:

wp-content, wp-includes, or any directory containing CSS and JavaScript. Google renders pages and needs the assets. Blocking them means Google sees an unstyled, non-functional page and cannot assess mobile usability. This is the most damaging one and it is very common.
wp-content/uploads. That is where your images live. Blocking it removes them from image search entirely and reduces the page’s understood content.
Category, tag, or archive pages, via robots.txt. If the goal is keeping thin pages out of the index, robots.txt is the wrong tool — see below.

The first is worth repeating because the reasoning changed and the advice did not. Blocking wp-content made sense when crawlers read HTML only. Now it actively harms rankings.

Why robots.txt does not remove pages from the index

This is the misconception that causes the most confusion, and it is worth being precise about.

robots.txt controls crawling, not indexing. Disallowing a URL tells crawlers not to fetch it. It does not tell them not to index it, and a URL that is linked from elsewhere can be indexed based on those links alone — appearing in results with no description, because the crawler was not permitted to read the page.

Worse, disallowing a URL prevents the crawler from seeing a noindex tag on it. So the standard mistake is:

Add a noindex tag to a page you want removed.
Also disallow it in robots.txt to be thorough.
The crawler never fetches the page, never sees the noindex, and the URL stays indexed indefinitely.

The correct approach: to keep a page out of the index, use a noindex meta tag or header and allow crawling so the directive can be read. To reduce crawler load on a section that should not be fetched at all, use robots.txt. They solve different problems and using both on the same URL is self-defeating.

What is actually worth blocking

A short, defensible list:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /*?replytocom=

Sitemap: https://example.com/wp-sitemap.xml

Each line earning its place:

wp-admin is the dashboard. No value in crawling it, and it requires authentication anyway.
admin-ajax.php must be allowed — plugins and themes use it for front-end functionality, and blocking it breaks rendering.
/?s= is internal search. Every search produces a URL, generating unlimited low-value pages if crawled.
replytocom is WordPress’s comment reply parameter. It creates a variant URL for every comment on every post, which can multiply a site’s URL count enormously.
The sitemap line helps discovery and costs nothing.

That is genuinely enough for most sites. If your robots.txt is thirty lines, it is worth asking what each line is for and whether it is still true.

When crawl budget matters

Crawl budget is invoked constantly and matters for very few sites. Google allocates crawling based on site size, update frequency, and server responsiveness, and for a site with a few hundred URLs it is not a constraint.

It becomes real for large sites, and the usual cause is URL multiplication rather than genuine page count:

Faceted navigation. Filter combinations producing thousands of URLs for the same products.
Session or tracking parameters creating a distinct URL per visitor.
Paginated archives running to hundreds of pages.
Attachment pages, which WordPress generates for every uploaded file.

The fix for most of these is not robots.txt. It is canonical tags pointing variants at the real URL, and for attachment pages, redirecting them to the parent post — a setting most SEO plugins expose.

Blocking faceted URLs in robots.txt is defensible when they are genuinely worthless and numerous. Do it with a parameter pattern rather than trying to enumerate paths, and check what you are matching before deploying it.

Checking it works

robots.txt errors are silent and can be expensive, so verify rather than assume.

Fetch it. curl https://example.com/robots.txt. Confirm it is what you expect and returns 200.
Test specific URLs in Google Search Console’s robots.txt tester, which tells you which rule matches a given URL. Precedence between Allow and Disallow is not intuitive and this removes the guesswork.
Check Search Console coverage for pages reported as blocked by robots.txt. Anything there that you wanted indexed is a live problem.
Re-check after any migration. A staging robots.txt disallowing everything, promoted to production, is one of the most common and most costly SEO accidents there is.

That last one is worth a calendar reminder after any launch. It is entirely silent — the site works perfectly and simply stops being crawled — and it is usually caught weeks later when traffic has already gone.

How this fits the rest of the stack

The pattern worth taking away is that robots.txt is a crawling control being used as an indexing control, and the two do not substitute for each other. Getting it wrong is silent, which is why the check after a launch matters more than the file itself. Managed WordPress on RunxBuild puts a file manager in the dashboard, so creating a real robots.txt or checking what is actually on disk does not start with SFTP credentials, and plans start at $3 a month. If you are sizing a site alongside its database and bandwidth, the RunxBuild hosting calculator itemises them.

Useful related references:

FAQ

Where is the robots.txt file in WordPress?

There is not one by default. WordPress generates it dynamically at request time. To change it, either create a real file at the site root — which overrides the virtual one — or use the robots_txt filter to keep it in version control.

Should I block wp-content in robots.txt?

No. Google renders pages and needs the CSS and JavaScript in wp-content to do so. Blocking it means Google sees an unstyled, non-functional page and cannot assess mobile usability. It also removes your images from image search.

Does robots.txt remove a page from Google?

No. It controls crawling, not indexing. A disallowed URL can still be indexed from links pointing at it, appearing with no description. Worse, blocking it prevents the crawler from ever seeing a noindex tag on the page.

How do I keep a page out of search results?

Use a noindex meta tag or header, and allow crawling so the directive can actually be read. Using both noindex and a robots.txt disallow on the same URL is self-defeating — the crawler never fetches the page to see the noindex.

Why did my site stop being indexed after launch?

Two usual causes: the search engine visibility setting in Reading settings is still discouraged from development, or a staging robots.txt disallowing everything was promoted to production. Both are silent and worth checking immediately after any launch.