How to Create and Optimize Robots.txt for SEO
The robots.txt file is one of the most important technical SEO elements. It tells search engine bots which pages they can crawl and which ones to avoid. By creating and optimizing this file correctly, you can improve your crawl budget, protect sensitive pages, and ensure search engines focus only on the content that matters.
What Is a Robots.txt File?
A robots.txt file is a small text file stored in your website’s root directory. It acts like a set of rules for search engine crawlers such as Googlebot or Bingbot.
Simple example:
User-agent: *
Disallow: /admin/
This tells all search engine bots not to crawl the /admin/ page.
Why Is Robots.txt Important for SEO?
A well-optimized robots.txt file helps you:
✔ Save crawl budget
Prevent Google from crawling unnecessary pages like filters, admin pages, or duplicate pages.
✔ Block sensitive content
Pages like checkout, login, or private sections should not appear in Google.
✔ Improve site performance
Bots skip unwanted pages, making crawling faster and more efficient.
✔ Control search engine behavior
You can allow, disallow, or point crawlers to important files like your XML sitemap.
Where Is the Robots.txt File Located?
It should always be available at:
https://yourwebsite.com/robots.txt
If this URL doesn’t load, the file is missing.
How to Create a Robots.txt File (Step-by-Step)
1. Open a simple text editor
Use Notepad, VSCode, or any basic editor.
2. Add crawling rules
Example:
User-agent: *
Disallow: /private/
Allow: /
3. Add your sitemap link
This helps Google crawl your website properly:
Sitemap: https://yourwebsite.com/sitemap.xml
4. Save and upload
Save the file as robots.txt and upload it to the root directory (public_html) using:
- cPanel File Manager
- FTP (like FileZilla)
- Hosting dashboard
Common Robots.txt Examples
1. Allow Everything (Recommended for Most Websites)
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
2. Block Specific Folders
User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Allow: /wp-admin/admin-ajax.php
3. Block Entire Website (Not Recommended for Live Sites)
User-agent: *
Disallow: /
4. Block Duplicate URLs or Filters
Disallow: /?sort=
Disallow: /?filter=
Best Practices to Optimize Robots.txt for SEO
✔ Do NOT block important pages
Avoid blocking category pages, product pages, or landing pages.
✔ Always include your sitemap
This helps search engines discover updated pages faster.
✔ Don’t use robots.txt to hide sensitive data
Use noindex, not robots.txt, for pages you don’t want shown in search.
✔ Test your file in Search Console
Google Search Console Robots.txt Tester helps you check errors.
✔ Keep rules simple
Unnecessary complexity can block important content by mistake.
Example of a Perfect Robots.txt for SEO
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /login/
Sitemap: https://example.com/sitemap.xml