Prevent Google And Other Search Engines From Indexing Certain Pages Of Your Blog

prevent

Image Credit: http://flic.kr/p/7vndX6

When we create a website or a Blog, Google automatically indexes many of our website pages. But many a times it happen that few pages are indexed by Google that we don’t want them to appear in search results.

Now What to do?

Many WordPress Themes give option to “noindex” pages, category, tags etc. Buy What about your theme files, admin files or files located here

 /wp-admin/
/wp-includes/
/wp-content/

You can’t control index and noindex of these directory or directory files from your WordPress dashboard.

So for removing these pages from Google Search result or any other search engine results we can create robots.txt file.

What is robots.txt file?

It is a simple text file that is created for giving special instructions to robots (like Googlebots). Whenever any robots comes for crawling your website, they check your robots.txt file for any special instruction to follow. This file is to be stored in root directory of your website.

How will it help?

The purpose of this file is exactly for telling Search Engines, what to index and what not. We just have to add some commands in robots.txt file for lowing or disallowing indexing of any page.

Steps to create Robots.txt file

1. Open notepad or any text editor.

2. Paste the following code if you are using WordPress and want to deindex WordPress files.

sitemap: http://example.com/sitemap.xml

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /go/
Disallow: /recommend/
Disallow: /archives/
disallow: /*?*
Disallow: /wp-*
Disallow: /author
Disallow: /comments/feed/

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

Replace above sitemap URL with your sitemap URL

3. Save the file as robots.txt

4. Upload file to the root directory of your website like example.com/robots.txt

5. That’s it, Next time when Search Engine will crawl your site these files or directories will be blocked from appearing in Search Engine.

Bonus: Create or Edit robots.txt file directly from WP dashboard

There is a Plugin available for WordPress than can help you to easily create or edit robots.txt file directly from your dashboard.

The Plugin is called KB Robots.txt

You can download this plugin from here

Upload it to the Plugin Directory and Activate it from WordPress Plugin page.

Now to Go to  ’Settings => KB Robots.txt’ admin page. In the box, write whatever you want your robots.txt file to say.

It’s a handy plugin that makes the whole process very easy and saves your time.

Let me know if you are stuck somewhere or have any query.

 

Leave a Reply

About Raj Sharma

Hey There! I am a passionate blogger and a web entrepreneur. I am addicted to internet and making money online. I am the founder of weppot.com and other amazing websites.

Leave a Reply