Comments on: WordPress Duplicate Content Prevention with robots.txt

By: Bogan Marketing

Bogan Marketing — Wed, 07 Apr 2010 12:34:34 +0000

nice info, there is detailed instructions on optimising robots.txt for seo and content duplication there > Robots.txt for silo SEO

By: Joe Jones

Joe Jones — Mon, 15 Feb 2010 00:12:37 +0000

Askvk,

I have one website that’s a wordpress site of only 2 web pages. I used all the disallows you have listed but Google’s keyword list for the site had “disallow” as the most frequently used keyword for the site. How do I stop google from using words in the robots.txt file as a source of keywords.

Joe Jones

By: VK

VK — Mon, 07 Dec 2009 21:04:28 +0000

In fact, the assumption is that the Archives and Categories listing just show the excerpt and not the full post content. Full post content will be only displayed when you are in front of a page or a post.

If you need even more control at the level of the SEO anti-duplicate settings, you’ll have to not only use the robots.txt file but also change other things at the level of the WordPress itself. For instance, using a SEO Plug-in or a SEO WordPress theme.

I have created a SEO WordPress theme that comes up with about 200 options/settings and lots of SEO options/settings among them to give the user a better control of the SEO aspect and the anti-duplicate content thing. If you have to look at my SEO WordPress theme, just go to URL:
http://www.vklabx.com/wordpress/

Then just read the PHP code provided with my SEO WordPress theme and if needed, just copy the code that may be of any interests to you and add/include that code to your WordPress site.

By: VK

VK — Mon, 07 Dec 2009 20:58:33 +0000

I understand what you mean. Sometimes using the robots.txt file only is not enough for smarter more dynamic web site. I was in such case too in the past and that’s why I finally created my SEO WordPress theme to help me add more anti-duplicate content feature directly via the WordPress Dashboard.

Please, when you have time, just quickly install my SEO WordPress theme located at URL:
http://www.vklabs.com/wordpress/

And you’ll see all the SEO anti-duplicate settings that you can play with. At first, it may be impressive but you’ll understand the logical goals.

By: VK

VK — Mon, 07 Dec 2009 20:48:20 +0000

Why not give a try to my SEO WordPress theme called AdFlex Niche/Blog. More info at URL:
http://www.vklabx.com/wordpress/

My SEO WordPress theme provides you with tons of SEO options and other settings that may help you get a higher on-page SEO score and so a better ranking in SE. Of course, on-page SEO is one important things but you also need to dedicate some times on the off-page SEO part.

Hope it helps.

By: Keith Davis

Keith Davis — Sat, 05 Dec 2009 11:27:17 +0000

Hi VK
I’m trying to put together a robots.txt file to prevent duplicate content and I notice that Archives and Categories are not excluded in your robots.txt file.

If they are not excluded, won’t they produce duplicate content?

By: Keith Davis

Keith Davis — Thu, 26 Nov 2009 18:08:09 +0000

Hi VK
Still trying to get my head around this duplicate content thing with WordPress, posts, archives, categories etc.
I can handle a static site, you just don’t repeat yourself, but with a dynamic site!
This certainly helps.

By: SEO Specialist

SEO Specialist — Wed, 07 Oct 2009 18:58:57 +0000

Great post and very informative. Google is emphasising more and more on uniqness of content and thus avoiding duplicacy could be of great help for SERP.

By: Internet Marketing Company

Internet Marketing Company — Tue, 14 Jul 2009 10:05:09 +0000

Wow, what a find. This post is definitely one of the best WP tips that I have seen. I also write a different excerpt for each post because it shows on the category page. This prevents duplicate content on the category pages. Keep up the good work!

By: Pagerank Checker

Pagerank Checker — Mon, 08 Jun 2009 18:04:01 +0000

You have a great blog here and it is Nice to read some well written posts that have some relevancy…keep up the good work