Wordpress offers many ways to handle your content, you can block certain URLs and URL paths, the rights robots.txt.One of the main reasons for creating a robots.txt file to prevent search engines in achieving Your capacity of more than one place (ie your monthly archives, category of your folders, your XML feed and on the front page), because this can lead to duplicate content issues.

Here’s my suggestion for the Wordpress robots.txt :

User-agent:  Googlebot

# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/

# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$

# Disallow parsing individual post feeds, categories and trackbacks..
Disallow: */trackback/
Disallow: */feed/
Disallow: /category/*

If you are thinking to avoid some more content like tags,wp-admin and trackbacks & also comments if you are tired of spam you can choose from following ones i am going for the tags

Disallow: /*trackback
Disallow: /*.css
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /tag/
Disallow: /author/
Disallow: /comments/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-admin/
Disallow: /*?*
Disallow: /*?

You can also use
http://www.joostdevalk.nl/wordpress/meta-robots-wordpress-plugin/ One option is to add a nofollow meta tags to header files in your “brand” directories. This will remove the duplicate content penalty as well.

Some Points to Remember

1. some devices are built Wordpress Gallery2 which – for reasons unknown – such as driving in main.php url (even with the url-rewriting enabled).
2. If your blog is in the sub-directory of your domain, and the robots.txt for the entire domain in mind that you could block a key site in the other sub-lists. Which is why robots.txt is not part of the default installation of WordPress.
   It is also necessary to ensure that the agents will be blocked, and it would be wise to specifically geared robots instead of using the problematic * symbol in the ‘user-agent field.

Popularity: 1% [?]

Related posts:

1. Seo guide to smf
2. smf forum and duplicate content
3. How to transfer Wordpress from subdirectory to root?
4. How to find post ID in wordpress
5. How To Login In Wordpress Using encrypted SSL
6. Must Have Wordpress Plugins Collection
7. Best Seo Plugins for Wordpress
8. How To Podcast Through Wordpress
9. Migrating >> B2Evolution To Wordpress
10. How to remove the Category Base from the url permalink structure Of Wordpress
11. wp-admin.php 404 not found Error Wordpress
12. How many Plugins you use in wordpress blog
13. Upgrade To wordpress 2.8 Failed
14. Automatic Blog Comment Poster
15. Free Latest Wordpress Themes
16. Customize your 404 page in Wordpress
17. How to Remove Malicious Footer Links Code In Wordpress
18. Unable to create directory – Is its parent directory writable by the server?
19. How To Automatically Resize Images In Wordpress Blogs
20. How To Install BBpress In Wordpress With Cookie Integration
21. How To Make Rss Feed Popular With Twitter
22. document.write unescape problem wordpress
23. How To remove wordpress “Just another wordpress blog”
24. How to Add Javascript Code In Wordpress Posts
25. How to create a wap or mobile sitemap ?