When Google wants to learn about your website, it sends a piece of software called a crawler (also known as a spider or a bot) to visit your pages and read their content. Before the crawler does anything else, it looks for a file called robots.txt at the very top level of your domain — the same level as your homepage. If that file exists, the crawler reads its instructions first.
The file uses a simple format: it names the crawler it is addressing (Google’s is called Googlebot) and then lists any pages or sections of the site it should not crawl. An instruction like Disallow: /admin/ tells all crawlers to stay out of the admin area of your site.
Why Would You Want to Block Google From Anything?
This might seem counterintuitive — why would you stop Google visiting your pages? There are a few legitimate reasons:
- Admin and login pages. Pages that only you as the website owner can see — your dashboard, settings area, and backend — do not need to appear in Google search results. Blocking them stops them showing up where they have no value to anyone searching.
- Thank-you pages and internal search results. If your website has a contact form, there is usually a thank-you page that appears after someone submits it. That page has no useful content for a new visitor arriving from Google, so it makes sense to keep it out of the search index.
- Staging or development versions of your site. If your web developer has a test version of your site on a subdomain while working on a redesign, you do not want that half-finished version appearing in Google results alongside your real site.
- Duplicate content. Some websites end up with multiple URLs showing the same page — with and without a trailing slash, or with different filter parameters. Blocking some of these variants prevents Google from seeing them as competing versions of the same content.
The One Thing Every Website Owner Should Check
When a website is being built or redesigned, most platforms offer an option to “hide from search engines” or “discourage search engines from indexing.” This is useful during development — you do not want a half-finished website appearing in Google results before it is ready. The platform achieves this by adding a disallow instruction to your robots.txt file that blocks all crawlers from all pages.
The problem is that this setting sometimes does not get switched off when the site goes live. The website looks completely normal to visitors — but Google is reading an instruction that says “stay away from everything.” The result: your site stops appearing in search results, and you may not notice for weeks or even months.
You can check your own robots.txt file right now. Type your domain name followed by /robots.txt into a browser address bar. If you see a line that reads Disallow: / with no restrictions around which crawler it applies to, that is a serious problem. A healthy robots.txt file for most small business websites either contains no disallow rules at all, or only blocks specific sections like /wp-admin/ — the WordPress backend.
Does Robots.txt Affect Your Google Ranking?
Not directly — a well-configured robots.txt file does not boost or reduce your rankings on its own. But it can cause a significant ranking problem if pages you want Google to index are being blocked. Think of it as a gate: the file controls which pages Google is allowed to look at in the first place, not how highly it ranks the pages it can see. If your important pages are behind a blocked gate, they cannot rank for anything at all.
It is also worth knowing that robots.txt only prevents crawling — not indexing entirely. This is a subtle but important distinction. If another website links to one of your blocked pages, Google may still list it in search results (because it heard about the page from the link) even if it cannot access the content. If you need a page to be completely invisible to Google, a noindex tag placed within the page’s HTML code is more reliable than a robots.txt block.
What Does a Typical Small Business Robots.txt Look Like?
For most small business websites, the robots.txt file is either very short or effectively empty. A healthy example for a WordPress site might look something like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.co.uk/sitemap.xml
The User-agent: * line means “these instructions apply to all crawlers.” The disallow line keeps Google out of the WordPress admin area. The allow line makes an exception for one technical file the site needs to function. And the sitemap line helpfully points Google toward the list of all your pages. If you see something similar when you check your own robots.txt, your site is in good shape.
What you do not want to see is Disallow: / on its own — the forward slash means “everything,” and that single line tells Google to crawl nothing at all on your website.