Walkthrough · Intermediate · 15 minutes
How to Write a robots.txt
Steer crawlers sensibly, and avoid the self-inflicted injuries this file is famous for — a hands-on walkthrough in plain words that names the classic mistake before you reach it.
The short answer
The job: steer crawlers sensibly, and avoid the self-inflicted injuries this file is famous for. Time to set aside: 15 minutes. Skill needed: intermediate.
Below is the full run of steps, the error most people trip over, and the short cut worth remembering. On Hosting & Domains plans several steps are already automated away — the guide marks each one.
By the Hosting & Domains team · Reviewed 18 August 2026
Intermediate
Experience required
5
Stages, start to finish
Free
Support included
Proven
On the platform itself
No technical background needed. This walkthrough was written for first-timers, proven on our own hardware, and honest about which parts are genuinely tricky rather than merely unfamiliar.
A promise before step one: nothing here is a one-way door. Any step with teeth is flagged, and the way back is printed beside it.
How the work divides up
The work falls into a few clean stages: understand what it is not, start almost empty, never disallow a page you want ranked, name the sitemap and validate it before relying on it.
No part of this needs code or a terminal unless the guide says so plainly — and where it does, the exact commands are printed. The step-by-step is below, with the surrounding sections supplying the context that makes it hold.
Where it comes off, and how to stay on
A Disallow: / left over from the build phase. One character walls the whole site off, the pages themselves look perfectly normal, and you find out when traffic flatlines. This is the launch-checklist entry that justifies having a checklist.
It earns a section of its own because it is no obscure edge case — it is the single commonest reason this task reaches a support queue. Knowing it in advance turns the whole job from risky into routine.
The habit that keeps this easy for good
Leave CSS and JavaScript paths unblocked. Google renders pages in full, so blocked assets make your pages look broken to it, which is a ranking penalty bought with a rule that gains you nothing.
Habits this small are what separate the people who find hosting effortless from the people who find it draining. The same tools on both sides — a different way of working.
Where this platform takes work off you
Steps that never deserved your time are taken out: certificates issue and reissue themselves, the installer handles application setup, the daily copy absorbs the what-ifs, and per-site settings live in a panel rather than a configuration file. The guide covers the remainder — the part that is actually about your site.
Should a step still misfire, support answers at any hour with an actual fix rather than a knowledge-base link and a shrug. Half the guides on this site began as repeat patterns in our ticket queue.
Why this job is shorter on our plans
Every walkthrough in this library is run on the platform we actually operate — cPanel, LiteSpeed, NVMe, one-click installs — so the instructions match your screen rather than gesturing at it from a distance.
Hosted somewhere else already? We move the whole site at no charge, usually within 24 hours, and it goes on answering visitors throughout.
- Every step checked exactly as published
- The snag named before it arrives
- The dull steps taken out of your way
- People reachable at any hour you stall
Why Hosting & Domains
Standard on every plan
5 steps, nothing padded
Each stage is a few minutes of steady clicking, and the fiddly moments are labelled fiddly.
The snag, flagged up front
The classic error for this particular task is named before step one, which is how 15 minutes stays 15 minutes.
The dull parts automated
SSL, backups and installs look after themselves here, leaving the guide to cover only what is genuinely yours.
Jargon never charges you
Every term is defined on the spot or linked to the jargon buster — nothing assumes prior knowledge.
Help that keeps your hours, not ours
Stuck on step three at midnight? Support answers at any hour, mid-walkthrough included.
Taken from real tickets
These guides came out of the support queue, so every snag flagged is one people genuinely hit.
Quick Start
Order placed to site online
- 1
Understand what it is not
It is crawl advice rather than access control. robots.txt asks well-behaved crawlers to keep away from certain paths; it hides nothing from a browser or from anybody acting in bad faith.
- 2
Start almost empty
Most sites need very little: allow everything, name the sitemap, and perhaps exclude genuine crawl waste such as internal search results or cart URLs.
- 3
Never disallow a page you want ranked
A blocked path cannot be crawled, so anything meant for the index has to stay reachable. The file's power is precisely why its mistakes cost so much.
- 4
Name the sitemap
The Sitemap: line puts the map inside the first file crawlers read. One line, and a standing invitation to explore.
- 5
Validate it before relying on it
Search Console's robots tester, along with loading /robots.txt directly, confirms the file says what you meant. Syntax errors fail silently, and they fail in the direction of over-blocking.
Built In
Fitted to every plan
- No set-up charge at any point, and no joining fee
- The name's first year included when you order annually
- SSH, Git and Composer on the developer plans
- Your existing site brought across by our engineers, at no charge
- A daily copy, with restores you run yourself from the panel
- Money back within 30 days on hosting plans, 7 on reseller
- 99.9% uptime as the target, watched around the clock
- Staging copies for trying a change before it goes live
- People on the support desk every hour of every day
- Softaculous included for one-click application installs
Frequently Asked
The questions that come up most
Will robots.txt keep private pages hidden?
It will not. It is a public file politely asking for distance, and it advertises every path it names. Anything genuinely private wants authentication behind it, whether a directory password or an application login. robots.txt directs traffic; it secures nothing.
Does every site need the file?
Sites run perfectly well without one, since a missing file simply means allow everything. Even so, a minimal file naming the sitemap is worth keeping, because owning that URL means you decide its contents rather than leaving them to chance.
Can I transfer in a name I already own?
Yes, and it is routine. Unlock the name at your current registrar, take the auth code, and start the transfer from your client area. Whatever registration time remains carries over, and DNS keeps resolving the whole way through.
Which control panel do accounts use?
cPanel, the panel the industry standardised on long ago. Every tutorial you find online will match what is on your screen, your backups restore onto any other cPanel host, and the skills stay useful for life. Plesk and DirectAdmin are available on particular plans if you prefer either.
Keep reading
How to Reduce Server Response Time
Shrink the pause before the very first byte departs the server — intermediate level, about an afternoon.
How to Change Nameservers
Hand a domain's DNS to a different provider with every record arriving in one piece — beginner level, about 5 minutes plus propagation.
TXT Record (Glossary)
A single term without the jargon: what it means, why it counts, and where it catches people out.
Joomla Hosting
Joomla in one click, with your existing site migrated in at no charge.
Web Hosting
cPanel hosting on NVMe drives — SSL, the migration and year one of the name all included.
Changing hosts? Run through our checklist first.
A plain order of work for a move nobody visiting will notice: which files go over first, how to carry the mail across without losing a message, when exactly to repoint DNS, and the two mistakes behind nearly every outage we are asked to rescue.
Start on a plan priced without games.
NVMe hosting with SSL and the migration included, people answering at any hour, and a price that stays where it was.
View Joomla Hosting plans