4
   

Prevent search engines from indexing your site by IP address

 
 
Reply Sat 10 Apr, 2004 09:28 pm
When search engines index your site by the IP address (e.g. http://67.15.36.14/~user/page.html ) instead of the domain name (e.g. http://www.able2know.org/page.html ) you can run into a LOT of problems.

  • Cookies might not work correctly.
  • If you move your site to a new IP address the old links might not reach your site.
  • But most importantly, the search engine might see your domain-based URLs as duplicate content and not index it.


Here is how to prevent/fix this. You need to be using the Apache web server and have mod_rewrite available.

In your .htaccess file, add this (using your own domain, of course):

Code:RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*able2know\.org[NC]
RewriteRule (.*) http://www.able2know.org/$1 [R=301,L]


Quote:
RewriteEngine on


This turns on the Rewrite Engine. If this already exists then skip it.

Quote:
RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]


This is the rewrite condition. It basically says that if the http host is not your domain to process the next rule.

Quote:
RewriteRule (.*) http://www.able2know.org/$1 [R=301,L]


This is the rewrite rule. It gives a search engine friendly 301 http redirect to the correct page.
  • Topic Stats
  • Top Replies
  • Link to this Topic
Type: Discussion • Score: 4 • Views: 359,427 • Replies: 52
No top replies

 
zoomsan
 
  1  
Reply Sun 11 Apr, 2004 12:40 pm
hello craven ..

i am just a tad bit confused, and this may be a simple question - but is the .htaccess file you talk about the one in the site's root directory? or the one in the forum's root directory? i assume it could be either ...

so, if i edited my .htaccess file in my sites root directory with:

RewriteRule (.*) http://www.able2know.com/$1 [R=301,L]

i assume that the rewrite would occur for all directories beyond the domain name?

the code or script portion of your post has:

RewriteRule (.*) http://www.able2know.com/forums/$1 [R=301,L]

with the /forums directory included which threw me off just a bit ...

i have more to my site then the forums so i assume it would be beneficial to implement this rule in my site's root directory .htaccess - is this logical?

also, would this "cover" my forums also?

finally, the contents of my current .htaccess file in my site's root is:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://---.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://---.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.---.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.---.com$ [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ http://www.---.com [R,NC]

can i simply add to this by adding your rewrite Cond, or rather your RewriteRule, after the rule that disables hotlinking that is listed here? I assume I can

This is my first time working with htaccess, but I have read a bit about it before and I know its important so I want to make sure I implement your great ideas properly. Actually I've heard that hotlinking maybe even screws up your site with utilities like Norton, etc - so maybe I am better off disabling it. What do you think?


thanks!

zm
0 Replies
 
Craven de Kere
 
  1  
Reply Sun 11 Apr, 2004 04:43 pm
Gotta do this quickly. So apologies for brevity.

You are correct that this is the file for my forums subdirectory.

I also have one in my root directory that says:

Code:RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]
RewriteRule (.*) http://www.able2know.com/$1 [R=301,L]


I keep both because for eventual load balancing they might be served off of a different server.

Norton can screw up hotlinking only because hotlinking prevention depends on the referrer string and the referrer string is alterable. Its support is continguent on the user/browser/other software and might not be correct. Utilities like Norton often block the referrer string as a part of their "privacy" suite. The result is that with some methods of hotlinking protection the images are denied.

I personally recommend placing the hotlink protection in the actual directories where there are images. This way the hotlink rewriting is only called into action for specific images (this way you can also have allowed images.

Here is a sample of what I use for hotlink protection:

Code:RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://able2know.com/.*$ [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ http://www.able2know.com/images/hotlink.jpe [R,NC,L]


Note the first line, it's intended to allow for an empty referrer string but because the referrer string can be set to "Happy happy joy joy" or any nonsense you can't predict them all.
0 Replies
 
UnXpected
 
  1  
Reply Thu 13 May, 2004 09:40 am
static URLs mod_rewrite !

It can't work ????
0 Replies
 
Craven de Kere
 
  1  
Reply Thu 13 May, 2004 09:50 am
What do you mean?
0 Replies
 
UnXpected
 
  1  
Reply Fri 14 May, 2004 05:32 am
for the .htaccess I need to create in my forum folder or Root ????
0 Replies
 
Craven de Kere
 
  1  
Reply Fri 14 May, 2004 09:18 am
It depends on what you want to do. This tutorial is for root.
0 Replies
 
MadNutt
 
  1  
Reply Wed 23 Jun, 2004 05:43 pm
Is there any type of final version of this available? I'm not too good in PHP just yet (spent the last 5 years in ASP and .NET), and after doing the mod I'm getting 404's.
0 Replies
 
Craven de Kere
 
  1  
Reply Wed 23 Jun, 2004 06:25 pm
That is functional code, and it's not PHP it's rewrite rules.
0 Replies
 
lavinya
 
  1  
Reply Sun 15 Aug, 2004 10:40 am
hi
hello .

RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]
RewriteRule (.*) http://www.able2know.com/forums/$1 [R=301,L]

RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*mysite\.com [NC]
RewriteRule (.*) http://www.mysite.com/phpBB2/$1 [R=301,L]

my forum directory is www.mysite.com/phpBB2

your code shows me that your forum directory is "http://www.able2know.com/forums/" will I change my forum directory from "phpBB2" to "forums" .

please tell me ..
0 Replies
 
lavinya
 
  1  
Reply Wed 18 Aug, 2004 10:02 am
Sad
0 Replies
 
Craven de Kere
 
  1  
Reply Wed 18 Aug, 2004 10:24 pm
I'm not sure what you are trying to ask.

But the example code I gave was not a mod, it's aimed at teaching the technique and being adapted.

The inclusion of the forum directory was not even intended.
0 Replies
 
gennaro
 
  1  
Reply Sat 16 Oct, 2004 11:36 pm
I am following the directions in the phpBB static URLs mod_rewrite 1.0.0 and am not clear about the htaccess or robots placement. Should they be in the main root directory, in the forum directory or both. Right now I have robots in the main one and the SEO 2.0.0 mod is working good.

But after setting up the static URLs mod my forum is all wack and I get "Parse error: parse error in /home/humandes/public_html/forum/includes/page_header.php on line 118" at the top of the page. So I'm obviously doing somthing wrong.

In the robot file it says to put many "Disallow: /forums/..." but my forum folder is "forum" (no 's'), so am i correct to take off the 's'?

Likewise, the htaccess info says "RewriteEngine On
RewriteRule ^forums.* index.php [L,NC]..."
Should I chance "forums" to "forum" here?

THanks
0 Replies
 
Craven de Kere
 
  1  
Reply Sat 16 Oct, 2004 11:39 pm
gennaro wrote:
I am following the directions in the phpBB static URLs mod_rewrite 1.0.0 and am not clear about the htaccess or robots placement. Should they be in the main root directory, in the forum directory or both.


robots.txt should always be in the root directory of a domain, as per the robots.txt standards. Spiders only look there for them.

The .htaccess code should be in the forum directory as per the mod instructions.

Quote:
In the robot file it says to put many "Disallow: /forums/..." but my forum folder is "forum" (no 's'), so am i correct to take off the 's'?


Yes

Quote:

Likewise, the htaccess info says "RewriteEngine On
RewriteRule ^forums.* index.php [L,NC]..."
Should I chance "forums" to "forum" here?

THanks


No, that is there just for testing purposes and is never used.
0 Replies
 
zoomsan
 
  1  
Reply Sat 30 Oct, 2004 09:55 pm
Hello,

I have a problem.

I've put this .htaccess mod in my root directory. It works fine. However, I have also placed an htaccess file in the forum's directory with the rules for the static page mod.

If I place the same mod for the NO-IP issue stated here in the forum directory, I obviously get shifted to the site home page - not the forum home page.

Further, if I change these no-ip rules to www\.site\.com/forum/ and www.site.com/fourm/$1 and add that to the top of the forum htaccess file I get an error for too many redirects, and it doesn't work.

The reason I still need to do that is because when I type site.com/fourm/ it does not redirect to www.site.com/forum/ - although site.com/forum obviously does.

Thanks in advance for any help
0 Replies
 
Craven de Kere
 
  1  
Reply Mon 1 Nov, 2004 03:48 pm
Why are you trying to put it in your forum directory? There's no need for that.
0 Replies
 
zoomsan
 
  1  
Reply Mon 1 Nov, 2004 05:14 pm
Craven de Kere wrote:
Why are you trying to put it in your forum directory? There's no need for that.
I should clarify and say that I have an .htaccess file in my forum directory with the mod_rewrite rules from the SEO mod. The problem comes in because the server sees this .htaccess file, without the re-direction rules (because as I stated in my last post I can't get them to work properly in the htaccess forum file) instead of the roots .htaccess file obviously when viewing files within the forum dir.

I could possibly put the mod_rewrite rules into the root directory, but I am not sure if they would work properly and also I was trying to keep the htaccess file small because I have other rewrite rules going on from just root.

Best case scenario would be to get an additional rule I guess for the forum htaccess file to go above the mod_rewrite rules.

I hope, together with my last post, my situation is more clear.

Thanks.
0 Replies
 
Craven de Kere
 
  1  
Reply Mon 1 Nov, 2004 08:17 pm
Here are my own rules in the forum directory, they work fine:

Code:RewriteEngine on
RewriteCond %{HTTP_HOST} !^.*able2know\.com [NC]
RewriteRule (.*) http://www.able2know.com/forums/$1 [R=301,L]

RewriteRule ^forums.* index.php
RewriteRule ^a2k-post([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?p=$1&highlight=$2 [L,NC]
RewriteRule ^a2k-post([0-9]*).* viewtopic.php?p=$1 [L]
RewriteRule ^a2k-view-poll([0-9]*)-([0-9]*)-([a-zA-Z]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&vote=viewresult [L,NC]
RewriteRule ^about([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?t=$1&highlight=$2 [L,NC]
RewriteRule ^about([0-9]*).html&view=newest viewtopic.php?t=$1&view=newest [L,NC]
RewriteRule ^about([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&start=$4 [L,NC]
RewriteRule ^about([0-9]*)-([0-9]*).* viewtopic.php?t=$1&start=$2 [L,NC]
RewriteRule ^about([0-9]*).* viewtopic.php?t=$1 [L,NC]
RewriteRule ^about([0-9]*).html viewtopic.php?t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 [L,NC]
RewriteRule ^mark-forum([0-9]*).html* viewforum.php?f=$1&mark=topics [L,NC]
RewriteRule ^updates-topic([0-9]*).html* viewtopic.php?t=$1&watch=topic [L,NC]
RewriteRule ^stop-updates-topic([0-9]*).html* viewtopic.php?t=$1&unwatch=topic [L,NC]
RewriteRule ^ask-about([0-9]*).html viewforum.php?f=$1 [L,NC]
RewriteRule ^ask-about([0-9]*).* viewforum.php?f=$1 [L,NC]
RewriteRule ^a2k-about([0-9]*)-([0-9]*)-([0-9]*).* viewforum.php?f=$1&topicdays=$2&start=$3 [L,NC]
RewriteRule ^ptopic([0-9]*).* viewtopic.php?t=$1&view=previous [L,NC]
RewriteRule ^ntopic([0-9]*).* viewtopic.php?t=$1&view=next [L,NC]


If you want to see them all in action: http://67.19.244.238/forums/about22856.html
0 Replies
 
zoomsan
 
  1  
Reply Tue 2 Nov, 2004 05:39 am
Actually your forum does the same thing I don't want mine to do. If you type in able2know.com/forums/ it does not automatically redirect to www.able2know.com/forums/ - this is what I am trying to achieve, you see.

I tried your rules here you just listed, which were the same, and it didn't do it. Any ideas? I dont want a pr split.
0 Replies
 
Craven de Kere
 
  1  
Reply Tue 2 Nov, 2004 09:33 am
Just add www to the condition. My rules are aimed at redirecting if an ip is used, so as long as it uses domain.com it's not enacted.

Just change the rule to specify www.domain.com
0 Replies
 
 

Related Topics

SEO - Going from 0 PR on up - Discussion by jespah
How Rich Pins work ??? - Question by OnceUponAtime
Need a bit of Google advice - Question by bible quizzer
Google Indexing Issue - Question by brainstudioz
Bing Search results - Discussion by iankyra
What is best way for Off-Page seo in 2017? - Question by honesttravel1
 
  1. Forums
  2. » Prevent search engines from indexing your site by IP address
Copyright © 2024 MadLab, LLC :: Terms of Service :: Privacy Policy :: Page generated in 0.1 seconds on 12/12/2024 at 10:36:29