Prevent search engines from indexing your site by IP address

Forums: Seo, Web Development, Cannonical Urls
Email this Topic • Print this Page

When search engines index your site by the IP address (e.g. http://67.15.36.14/~user/page.html ) instead of the domain name (e.g. http://www.able2know.org/page.html ) you can run into a LOT of problems.

Cookies might not work correctly.
If you move your site to a new IP address the old links might not reach your site.
But most importantly, the search engine might see your domain-based URLs as duplicate content and not index it.

Here is how to prevent/fix this. You need to be using the Apache web server and have mod_rewrite available.

In your .htaccess file, add this (using your own domain, of course):

Code:

RewriteEngine on

RewriteCond %{HTTP_HOST} !^.*able2know\.org[NC]

RewriteRule (.*) http://www.able2know.org/$1 [R=301,L]

Quote:

RewriteEngine on

This turns on the Rewrite Engine. If this already exists then skip it.

Quote:

RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]

This is the rewrite condition. It basically says that if the http host is not your domain to process the next rule.

Quote:

RewriteRule (.*) http://www.able2know.org/$1 [R=301,L]

This is the rewrite rule. It gives a search engine friendly 301 http redirect to the correct page.

Topic Stats
Top Replies
Link to this Topic

Type: Discussion • Score: 4 • Views: 359,176 • Replies: 52

No top replies

hello craven ..

i am just a tad bit confused, and this may be a simple question - but is the .htaccess file you talk about the one in the site's root directory? or the one in the forum's root directory? i assume it could be either ...

so, if i edited my .htaccess file in my sites root directory with:

RewriteRule (.*) http://www.able2know.com/$1 [R=301,L]

i assume that the rewrite would occur for all directories beyond the domain name?

the code or script portion of your post has:

RewriteRule (.*) http://www.able2know.com/forums/$1 [R=301,L]

with the /forums directory included which threw me off just a bit ...

i have more to my site then the forums so i assume it would be beneficial to implement this rule in my site's root directory .htaccess - is this logical?

also, would this "cover" my forums also?

finally, the contents of my current .htaccess file in my site's root is:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://---.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://---.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.---.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.---.com$ [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ http://www.---.com [R,NC]

can i simply add to this by adding your rewrite Cond, or rather your RewriteRule, after the rule that disables hotlinking that is listed here? I assume I can

This is my first time working with htaccess, but I have read a bit about it before and I know its important so I want to make sure I implement your great ideas properly. Actually I've heard that hotlinking maybe even screws up your site with utilities like Norton, etc - so maybe I am better off disabling it. What do you think?

thanks!

zm

0 Replies

Gotta do this quickly. So apologies for brevity.

You are correct that this is the file for my forums subdirectory.

I also have one in my root directory that says:

Code:

RewriteEngine on 

RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]

RewriteRule (.*) http://www.able2know.com/$1 [R=301,L]

I keep both because for eventual load balancing they might be served off of a different server.

Norton can screw up hotlinking only because hotlinking prevention depends on the referrer string and the referrer string is alterable. Its support is continguent on the user/browser/other software and might not be correct. Utilities like Norton often block the referrer string as a part of their "privacy" suite. The result is that with some methods of hotlinking protection the images are denied.

I personally recommend placing the hotlink protection in the actual directories where there are images. This way the hotlink rewriting is only called into action for specific images (this way you can also have allowed images.

Here is a sample of what I use for hotlink protection:

Code:

RewriteCond %{HTTP_REFERER} !^$

RewriteCond %{HTTP_REFERER} !^http://able2know.com/.*$      [NC]

RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ http://www.able2know.com/images/hotlink.jpe [R,NC,L]

Note the first line, it's intended to allow for an empty referrer string but because the referrer string can be set to "Happy happy joy joy" or any nonsense you can't predict them all.

0 Replies

static URLs mod_rewrite !

It can't work ????

0 Replies

What do you mean?

0 Replies

for the .htaccess I need to create in my forum folder or Root ????

0 Replies

It depends on what you want to do. This tutorial is for root.

0 Replies

Is there any type of final version of this available? I'm not too good in PHP just yet (spent the last 5 years in ASP and .NET), and after doing the mod I'm getting 404's.

0 Replies

That is functional code, and it's not PHP it's rewrite rules.

0 Replies

hi
hello .

RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]
RewriteRule (.*) http://www.able2know.com/forums/$1 [R=301,L]

RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*mysite\.com [NC]
RewriteRule (.*) http://www.mysite.com/phpBB2/$1 [R=301,L]

my forum directory is www.mysite.com/phpBB2

your code shows me that your forum directory is "http://www.able2know.com/forums/" will I change my forum directory from "phpBB2" to "forums" .

please tell me ..

0 Replies

0 Replies

I'm not sure what you are trying to ask.

But the example code I gave was not a mod, it's aimed at teaching the technique and being adapted.

The inclusion of the forum directory was not even intended.

0 Replies

I am following the directions in the phpBB static URLs mod_rewrite 1.0.0 and am not clear about the htaccess or robots placement. Should they be in the main root directory, in the forum directory or both. Right now I have robots in the main one and the SEO 2.0.0 mod is working good.

But after setting up the static URLs mod my forum is all wack and I get "Parse error: parse error in /home/humandes/public_html/forum/includes/page_header.php on line 118" at the top of the page. So I'm obviously doing somthing wrong.

In the robot file it says to put many "Disallow: /forums/..." but my forum folder is "forum" (no 's'), so am i correct to take off the 's'?

Likewise, the htaccess info says "RewriteEngine On
RewriteRule ^forums.* index.php [L,NC]..."
Should I chance "forums" to "forum" here?

THanks

0 Replies

gennaro wrote:

I am following the directions in the phpBB static URLs mod_rewrite 1.0.0 and am not clear about the htaccess or robots placement. Should they be in the main root directory, in the forum directory or both.

robots.txt should always be in the root directory of a domain, as per the robots.txt standards. Spiders only look there for them.

The .htaccess code should be in the forum directory as per the mod instructions.

Quote:

In the robot file it says to put many "Disallow: /forums/..." but my forum folder is "forum" (no 's'), so am i correct to take off the 's'?

Yes

Quote:

Likewise, the htaccess info says "RewriteEngine On
RewriteRule ^forums.* index.php [L,NC]..."
Should I chance "forums" to "forum" here?

THanks

No, that is there just for testing purposes and is never used.

0 Replies

Hello,

I have a problem.

I've put this .htaccess mod in my root directory. It works fine. However, I have also placed an htaccess file in the forum's directory with the rules for the static page mod.

If I place the same mod for the NO-IP issue stated here in the forum directory, I obviously get shifted to the site home page - not the forum home page.

Further, if I change these no-ip rules to www\.site\.com/forum/ and www.site.com/fourm/$1 and add that to the top of the forum htaccess file I get an error for too many redirects, and it doesn't work.

The reason I still need to do that is because when I type site.com/fourm/ it does not redirect to www.site.com/forum/ - although site.com/forum obviously does.

Thanks in advance for any help

0 Replies

Why are you trying to put it in your forum directory? There's no need for that.

0 Replies

Craven de Kere wrote:

Why are you trying to put it in your forum directory? There's no need for that.

I should clarify and say that I have an .htaccess file in my forum directory with the mod_rewrite rules from the SEO mod. The problem comes in because the server sees this .htaccess file, without the re-direction rules (because as I stated in my last post I can't get them to work properly in the htaccess forum file) instead of the roots .htaccess file obviously when viewing files within the forum dir.

I could possibly put the mod_rewrite rules into the root directory, but I am not sure if they would work properly and also I was trying to keep the htaccess file small because I have other rewrite rules going on from just root.

Best case scenario would be to get an additional rule I guess for the forum htaccess file to go above the mod_rewrite rules.

I hope, together with my last post, my situation is more clear.

Thanks.

0 Replies

Here are my own rules in the forum directory, they work fine:

Code:

RewriteEngine on

RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]

RewriteRule (.*) http://www.able2know.com/forums/$1 [R=301,L] 



RewriteRule ^forums.* index.php  

RewriteRule ^a2k-post([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?p=$1&highlight=$2 [L,NC]

RewriteRule ^a2k-post([0-9]*).* viewtopic.php?p=$1 [L]

RewriteRule ^a2k-view-poll([0-9]*)-([0-9]*)-([a-zA-Z]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&vote=viewresult [L,NC]

RewriteRule ^about([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?t=$1&highlight=$2 [L,NC]

RewriteRule ^about([0-9]*).html&view=newest viewtopic.php?t=$1&view=newest [L,NC]

RewriteRule ^about([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&start=$4 [L,NC] 

RewriteRule ^about([0-9]*)-([0-9]*).* viewtopic.php?t=$1&start=$2 [L,NC]

RewriteRule ^about([0-9]*).* viewtopic.php?t=$1 [L,NC]

RewriteRule ^about([0-9]*).html viewtopic.php?t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 [L,NC]

RewriteRule ^mark-forum([0-9]*).html* viewforum.php?f=$1&mark=topics [L,NC]

RewriteRule ^updates-topic([0-9]*).html* viewtopic.php?t=$1&watch=topic [L,NC]

RewriteRule ^stop-updates-topic([0-9]*).html* viewtopic.php?t=$1&unwatch=topic [L,NC]

RewriteRule ^ask-about([0-9]*).html viewforum.php?f=$1 [L,NC]

RewriteRule ^ask-about([0-9]*).* viewforum.php?f=$1 [L,NC]

RewriteRule ^a2k-about([0-9]*)-([0-9]*)-([0-9]*).* viewforum.php?f=$1&topicdays=$2&start=$3 [L,NC]

RewriteRule ^ptopic([0-9]*).* viewtopic.php?t=$1&view=previous [L,NC]

RewriteRule ^ntopic([0-9]*).* viewtopic.php?t=$1&view=next [L,NC]

If you want to see them all in action: http://67.19.244.238/forums/about22856.html

0 Replies

Actually your forum does the same thing I don't want mine to do. If you type in able2know.com/forums/ it does not automatically redirect to www.able2know.com/forums/ - this is what I am trying to achieve, you see.

I tried your rules here you just listed, which were the same, and it didn't do it. Any ideas? I dont want a pr split.

0 Replies

Just add www to the condition. My rules are aimed at redirecting if an ip is used, so as long as it uses domain.com it's not enacted.

Just change the rule to specify www.domain.com

0 Replies

Prevent search engines from indexing your site by IP address

Related Topics

Quick Links

My Account

able2know