WordPress

WordPress Robots.txt Optimization

The purpose of the robots.txt file is to direct search engines to where they can index and where they are not allowed to index.

It is recommended by all search engines to use this robots.txt file so that they don't waste time and resources crawling and indexing files which you don't want them to be indexed.

Here is what WordPress recommends your robots.txt file to look like.

Sitemap: http://www.example.com/sitemap.xml

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

# digg mirror
User-agent: duggmirror
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /app/uploads/
Back to top

Fastest WordPress Hosting With WPEngine

Stunning speed, powerful security, and best-in-class customer service. At WP Engine.

Risk free for 60 days