Your Shopify robots.txt is blocking products from Google — and the admin will never tell you

Short answer. Shopify generates /robots.txt from a robots.txt.liquid template that lives in your theme, so an app, an agency or a past developer can add a Disallow rule that stops Google fetching your product pages. Nothing in Shopify admin reports it. Read your live robots.txt, then check Search Console's Page indexing report for "Blocked by robots.txt".

A crawl block is a strange kind of outage. The product page loads perfectly for you, with a price, an add-to-cart button and a green "Active" label in Shopify admin. Google simply stops asking for it — and because Google never reports an error back to your store, nothing in the admin says that a whole path of your catalogue has gone dark.

The rule doing this is usually a single line, in a file most merchants do not know they own.

Blocked from crawling is not the same as blocked from indexing

These two get used interchangeably and they are not the same mechanism, which is why the fix so often goes wrong.

A noindex rule says: fetch this page, then keep it out of the index. A robots.txt Disallow says something much blunter: do not fetch this page at all. The second one sounds stronger. It is actually weaker, because a page Google never fetches is a page Google knows nothing about — including whether you wanted it indexed.

Google's Search Console documentation splits this into two separate statuses, and the difference between them is the whole story. "Blocked by robots.txt" means the URL was blocked, and Google notes that this does not guarantee the page will stay out of the index. "Indexed, though blocked by robots.txt" means it got in anyway: Google will not request and crawl the page, but can still index it using information from the pages that link to it.

For a product page that second status is close to the worst available outcome. The URL can surface in results, but Google has no title, no description, no price and no structured data to build a listing from — because it was never allowed to read any of it.

Why did this happen if nobody touched robots.txt?

Because on Shopify, robots.txt is not a static file you upload once. It is generated from robots.txt.liquid, a template that sits in the templates folder of your theme. Shopify's own documentation for editing robots.txt.liquid is unusually blunt about the stakes: it calls the file an unsupported customization, says Shopify Support cannot help with edits to it, recommends working with a Shopify Partner or having genuine code and SEO expertise, and states plainly that incorrect use of the feature can result in loss of all traffic.

Anything with theme code access can write to it: an SEO app you installed and forgot about, an agency hardening the file during a migration, or a well-intentioned edit made two themes ago.

A merchant on the Shopify Community posted exactly this pattern in March 2025: Search Console reported 34 blocked pages, and the merchant's own reading was that the cause was theme changes made by three apps they had installed. That is the characteristic shape of the failure. Nobody edited a file. Somebody installed something.

Shopify ships sensible defaults here — /admin, /cart, /checkout, /search, /policies/ and filtered collection URLs are disallowed out of the box, and those should stay that way. The damage comes from what gets added underneath them.

Want to know what Google is actually allowed to fetch on your store? We check your live catalogue against what your store publicly serves and flag product URLs Google cannot reach — read-only, no admin access, about 2 minutes. → free scan

The trap: a noindex Google is never allowed to read

The two mechanisms collide here, and it catches experienced people.

Suppose a page was blocked in robots.txt and it is now showing as "Indexed, though blocked". The instinctive fix is to add a noindex tag. It will not work, and Google says so directly: for a noindex rule to be honoured, the page must not be blocked by robots.txt, because a blocked crawler never sees the rule. The page stays exactly where it was.

The correct order is counter-intuitive: unblock first, so Google can come and read the instruction you want it to obey. This is the mirror image of the failure we covered in the phantom noindex article — there, Google was allowed in and told to forget the page; here, Google is not allowed in at all, so it never gets told anything. For reference on how common the other one is: noindex on product pages showed up on 85 of the 822 Shopify stores we have scanned end to end, 79 of those at critical severity.

What it costs inside Merchant Center

Organic search is only half of it. Google Merchant Center re-crawls the landing page behind every item in your feed, and it documents a specific failure for pages it cannot reach: product landing pages that robots.txt forbids Google's crawler to download. Google's stated consequence is not a warning — those products remain disapproved and stop showing up in your Shopping ads and free product listings until the crawler can reach the page again.

The same page names the two user agents that matter: Googlebot for the landing pages and Googlebot-image for the images. Both need access. A rule that only blocks your image CDN path is enough to break the second one while the first still looks fine.

One subtlety before you edit anything: Google's robots.txt specification resolves conflicts by the most specific rule — the longest matching path wins — and scopes every rule to a single host, protocol and port, so a rule on your primary domain says nothing about a separate CDN hostname.

How to check your own store in five minutes

1. Read the live file. Open https://yourstore.com/robots.txt in a browser. It is public — you do not need any tool or login. Read the Disallow lines and ask whether any of them could match a product URL.
2. Open Search Console → Page indexing. Look for the statuses "Blocked by robots.txt" and "Indexed, though blocked by robots.txt". Each one lists the affected URLs, which beats reasoning about wildcards in your head.
3. Run URL Inspection on one real product URL you expect to rank. It reports whether the crawl was allowed, instead of leaving you to infer it from a pattern.
4. Look for the template. Online Store → Themes → Edit code on the published theme → templates. If robots.txt.liquid is listed, your store is running a customized file and it is worth reading in full.
5. Check Merchant Center diagnostics for product-page crawl issues, which will name the URLs Google could not download.

The fix

The repair itself is short. The verification matters as much as the edit.

• Remove or narrow the rule in robots.txt.liquid on the published theme. Leave Shopify's defaults for /cart, /checkout and /admin alone; they are there for good reasons.
• Check the group that actually applies to Googlebot. A named User-agent block replaces the wildcard block entirely for that crawler, so a permissive * group does not rescue a restrictive Googlebot group.
• Do not wait for an instant result. Google generally caches robots.txt for up to 24 hours, and Merchant Center states a change may take 24 to 72 hours to reflect. Re-checking ten minutes later and seeing no change proves nothing.
• If you genuinely wanted those pages out of the index, unblock first, then add noindex — in that order, for the reason above.

Why this one comes back

Fixing the rule takes a few minutes. Keeping it fixed is the harder problem.

robots.txt.liquid belongs to a theme, not to the store, so whichever theme is published decides what Google is served. Shopify documents that uploading a theme from the Themes section of the admin will not import robots.txt.liquid, while ThemeKit or command-line changes preserve it — the file's presence depends on how a theme arrived, and swapping the published theme can quietly change the rules Google reads. Add an app with theme code access or a developer restoring a backup, and the same line can return months later.

None of it produces a symptom in the admin. It is the same silence as a product dropping out of your sitemap or a deleted product lingering in Google's results: the store is fine, the data is fine, and the failure is only visible on Google's side — where nobody is looking daily.