WordPress offers many ways to handle your content, you can block certain URLs and URL paths, the rights robots.txt.One of the main reasons for creating a robots.txt file to prevent search engines in achieving Your capacity of more than one place (ie your monthly archives, category of your folders, your XML feed and on the front page), because this can lead to duplicate content issues.
Here’s my suggestion for the WordPress robots.txt :
User-agent: Googlebot
# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
# Disallow parsing individual post feeds, categories and trackbacks..
Disallow: */trackback/
Disallow: */feed/
Disallow: /category/*
If you are thinking to avoid some more content like tags,wp-admin and trackbacks & also comments if you are tired of spam you can choose from following ones i am going for the tags
Disallow: /*trackback
Disallow: /*.css
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /tag/
Disallow: /author/
Disallow: /comments/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-admin/
Disallow: /*?*
Disallow: /*?
You can also use
http://www.joostdevalk.nl/wordpress/meta-robots-wordpress-plugin/ One option is to add a nofollow meta tags to header files in your “brand” directories. This will remove the duplicate content penalty as well.
Some Points to Remember
- some devices are built WordPress Gallery2 which – for reasons unknown – such as driving in main.php url (even with the url-rewriting enabled).
- If your blog is in the sub-directory of your domain, and the robots.txt for the entire domain in mind that you could block a key site in the other sub-lists. Which is why robots.txt is not part of the default installation of WordPress.
It is also necessary to ensure that the agents will be blocked, and it would be wise to specifically geared robots instead of using the problematic * symbol in the ‘user-agent field.
