robots.txt
robots.txt is a plain-text file at a website's root that tells search engine crawlers which URLs or sections of the site they may and may not request.
Last updated: June 9, 2026
robots.txt follows the Robots Exclusion Protocol, using simple Allow and Disallow rules to steer crawler behavior. It's a request rather than a lock, but reputable crawlers like Googlebot respect it, which makes it a standard way to manage how a site gets crawled.
A Shopify store often disallows internal search and cart URLs so crawlers spend their time on real product and collection pages, and it usually links to the XML sitemap as well. One important catch: robots.txt controls crawling, not indexing, so it isn't a way to hide a page from results. A URL blocked here can still appear in search if it's linked elsewhere; to keep a page out, use a noindex tag instead.
Used with care, robots.txt protects crawl budget by keeping bots away from low-value or duplicate URLs and pointing them at the content you actually want found.
Related terms
Alt text
Alt text is a short written description of an image, added in the HTML, that screen readers read aloud to people who can't see the image and that search engines use to understand what the picture shows.
Backlink
A backlink is a link from another website that points to a page on yours. Search engines read backlinks from trusted sites as a signal that your content is credible, which can lift your rankings.
Canonical URL
A canonical URL is the version of a page you tell search engines to treat as the main one when the same or similar content can be reached through several different URLs. It points them to the copy you want indexed and ranked.
Core Web Vitals
Core Web Vitals are three Google metrics that measure how a page actually feels to use: how fast the main content loads, how quickly the page responds when you interact with it, and how much the layout jumps around while loading.
Share this Story