Comments on: WordPress Duplicate Content Prevention with robots.txt

By: Droid Tv

Droid Tv — Sat, 26 Mar 2011 04:19:39 +0000

@ Al Sefati: Where did you hear that Robots.txt was made for, and ONLY for, preventing access to “inside pages such as login and admin”?

Robots.txt was made to prevent access to any pages the webmaster does not want indexed is what I have always heard.

@jon: this wont “help” with your rankings, but it will avoid you losing rank due to google finding duplicate content on your site and assuming your site is stealing or replicating content.

By: jon

jon — Sun, 28 Nov 2010 22:22:40 +0000

Has anyone tested and seen that this helps with their rankings? just wondering if it is worth it
thanks
jon

By: Al Sefati

Al Sefati — Wed, 10 Nov 2010 00:48:24 +0000

Forgive me if I disagree. I don’t really subscribe to blocking pages by robots.txt for purpose of duplicated content prevention. That is not why robots.txt made for.

Robots.txt was made to prevent search engines accessing inside pages such as login and admin pages. Any other usages of Robots.txt is incorrect.

I recommend using rel canonical link for the purpose you mentioned.

By: play solitaire online

play solitaire online — Thu, 09 Sep 2010 00:12:16 +0000

Thanks for sharing the link, but unfortunately it seems to be down… Does anybody have a mirror or another source? Please reply to my post if you do!

I would appreciate if a staff member here at http://www.askvk.com could post it.

Thanks,
Alex

By: Bogan Marketing

Bogan Marketing — Wed, 07 Apr 2010 12:34:34 +0000

nice info, there is detailed instructions on optimising robots.txt for seo and content duplication there > Robots.txt for silo SEO

By: Joe Jones

Joe Jones — Mon, 15 Feb 2010 00:12:37 +0000

Askvk,

I have one website that’s a wordpress site of only 2 web pages. I used all the disallows you have listed but Google’s keyword list for the site had “disallow” as the most frequently used keyword for the site. How do I stop google from using words in the robots.txt file as a source of keywords.

Joe Jones

By: VK

VK — Mon, 07 Dec 2009 21:04:28 +0000

In fact, the assumption is that the Archives and Categories listing just show the excerpt and not the full post content. Full post content will be only displayed when you are in front of a page or a post.

If you need even more control at the level of the SEO anti-duplicate settings, you’ll have to not only use the robots.txt file but also change other things at the level of the WordPress itself. For instance, using a SEO Plug-in or a SEO WordPress theme.

I have created a SEO WordPress theme that comes up with about 200 options/settings and lots of SEO options/settings among them to give the user a better control of the SEO aspect and the anti-duplicate content thing. If you have to look at my SEO WordPress theme, just go to URL:
http://www.vklabx.com/wordpress/

Then just read the PHP code provided with my SEO WordPress theme and if needed, just copy the code that may be of any interests to you and add/include that code to your WordPress site.

By: Keith Davis

Keith Davis — Sat, 05 Dec 2009 11:27:17 +0000

Hi VK
I’m trying to put together a robots.txt file to prevent duplicate content and I notice that Archives and Categories are not excluded in your robots.txt file.

If they are not excluded, won’t they produce duplicate content?

By: Keith Davis

Keith Davis — Thu, 26 Nov 2009 18:08:09 +0000

Hi VK
Still trying to get my head around this duplicate content thing with WordPress, posts, archives, categories etc.
I can handle a static site, you just don’t repeat yourself, but with a dynamic site!
This certainly helps.

By: SEO Specialist

SEO Specialist — Wed, 07 Oct 2009 18:58:57 +0000

Great post and very informative. Google is emphasising more and more on uniqness of content and thus avoiding duplicacy could be of great help for SERP.