Robots.txt
A note to crawlers about where not to go. Easy to leave in the wrong state after a draft.
What does Robots.txt mean?
A small public file that tells well-behaved crawlers which parts of a site they should not request. It is not a lock, and it does not remove a page already in search.
Also called: robots file
A courtesy list
The file usually lives at /robots.txt. It can name folders or pages that a crawler should skip, and it can point at the XML sitemap. Serious search engines read it. A person with the URL can still open the page. Do not put secrets behind robots.txt.
A noindex instruction, which is a different mechanism, asks Google not to keep a page in search. Robots.txt only asks crawlers not to fetch. Mixing the two up is how owners think they have hidden a page that is still readable, or blocked a page they meant to hide from results only.
The draft-site mistake
Staging tools and some builders add a rule that blocks the entire site while you design. If that rule ships to the public domain, Google is asked to stay away from a site you just launched. Search Console will eventually show the exclusion. The file itself takes ten seconds to open in a browser.
Check it on launch morning. You want the public pages allowed, the sitemap mentioned if you have one, and any true junk, such as an internal search results URL, discouraged. The launch checklist is the wider list. This file is one line of it.
What not to block
Do not block CSS or the files that make the page readable if you want Google to see the page as a person sees it. Do not block the whole site to keep one thank-you page quiet. Deal with that page on its own.
We review the file before a domain points at a new Artol site. If you add a tool later that rewrites robots.txt, tell us. A plugin update can restore a block you already removed.
An example
A mover's new site looked finished, and a search for the company name still showed only the old builder. The public robots.txt disallowed every page, left over from a staging toggle. Removing those two lines let crawlers request the site. It did not assign a rank. It removed a padlock they had not meant to keep.
Questions we get asked
Will robots.txt stop a scraper?
No. Rude tools ignore it. Use it for search engines you want to cooperate with. Use a password or a private host for anything that must stay unlisted.
Should I block pages I no longer offer?
Prefer a redirect to the closest current page, or a gone status if nothing replaced them. Blocking the crawler leaves the old URL unexplained. Redirect is the better tool for a retired service.
