2
   

phpBB robots.txt tutorial

 
 
Craven de Kere
 
  1  
Reply Wed 30 Mar, 2005 10:12 am
It's for the mod_rewrite portion of the mod.
0 Replies
 
graziano
 
  1  
Reply Thu 5 May, 2005 01:04 am
LocanT wrote:
Craven or anyone else

Quote:
Disallow: forums/updates-topic.html*$
Disallow: forums/stop-updates-topic.html*$
Disallow: forums/ptopic*.html$
Disallow: forums/ntopic*.html$


are these file specific?

In other words if my forum is named myforumsshould they be in my robots.text

Disallow: myforums/updates-topic.html*$
Disallow: myforums/stop-updates-topic.html*$
Disallow: myforums/ptopic*.html$
Disallow: myforums/ntopic*.html$

Thank you very much in advance



is it not more correct this ? (look at / before myforums)

Disallow: /myforums/updates-topic.html*$
Disallow: /myforums/stop-updates-topic.html*$
Disallow: /myforums/ptopic*.html$
Disallow: /myforums/ntopic*.html$
0 Replies
 
Craven de Kere
 
  1  
Reply Thu 5 May, 2005 01:13 am
yes
0 Replies
 
graziano
 
  1  
Reply Thu 5 May, 2005 01:39 am
do you think is better to use for example

Disallow: /myforum/search.php

or (it's better to use ...) this

Disallow: /myforum/search.php*


Thank you
Graziano
0 Replies
 
Craven de Kere
 
  1  
Reply Thu 5 May, 2005 01:41 am
wildcards are not standard.
0 Replies
 
graziano
 
  1  
Reply Thu 5 May, 2005 01:44 am
I asked it because you used them on

Disallow: /myforums/ntopic*.html$
0 Replies
 
Craven de Kere
 
  1  
Reply Thu 5 May, 2005 01:49 am
At one time I thought it would be adopted, but it hasn't picked up.

regex expressions would be very useful in robots.txt but adoption of them as a standard doesn't seem imminent.

In this thread I have given examples of how to work around this limitation.
0 Replies
 
phenforum
 
  1  
Reply Mon 27 Jun, 2005 02:12 pm
Hi everyone,

I was wondering if anyone knows why my phpBB forum has been
suddenly dropped form the MSN index.

Edit (Moderator): Link removed

Could it be my robots.txt? What about my mod_rewrite?
Could my keyword url html pages have been red-flagged?
0 Replies
 
SFC
 
  1  
Reply Mon 18 Jul, 2005 09:56 pm
Do I have this right for my forums:

User-agent: *
Disallow: /forums/admin/
Disallow: /forums/db/
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/language/
Disallow: /forums/templates/
Disallow: /forums/common.php
Disallow: /forums/config.php
Disallow: /forums/groupcp.php
Disallow: /forums/memberlist.php
Disallow: /forums/posting.php
Disallow: /forums/profile.php
Disallow: /forums/printview.php
Disallow: /forums/privmsg.php
Disallow: /forums/viewonline.php
Disallow: /forums/faq.php
Disallow: /forums/ptopic
Disallow: /forums/ntopic


Looks like google is still indexing mysite.com/forums/post.html and topic.html

and not the forums/about.html links?

Thanks for the help
0 Replies
 
Craven de Kere
 
  1  
Reply Tue 19 Jul, 2005 08:31 pm
You don't have any rules to prevent indexing posts in your above post, however you do have it now on your robots.txt.

So if that was a recent addition expect it to take effect shortly.

If you want to hasten the process you can use search engines' removal tools.

For example, Google's is here:

http://services.google.com:8882/urlconsole/controller?cmd=reload&lastcmd=login

More information can be found here:

http://www.google.com/remove.html
0 Replies
 
SFC
 
  1  
Reply Wed 20 Jul, 2005 07:20 am
Thanks I did recenlty add it after re-reading this post...

I'll keep you posted...

Mike
0 Replies
 
turna
 
  1  
Reply Fri 5 Aug, 2005 09:37 am
hello ,i use this robot.txt. i wonder everything ok or not?


Code:User-agent: *
Disallow: /forum/admin/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/templates/
Disallow: /forum/common.php
Disallow: /forum/groupcp.php
Disallow: /forum/memberlist.php
Disallow: /forum/modcp.php
Disallow: /forum/posting.php
Disallow: /forum/profile.php
Disallow: /forum/privmsg.php
Disallow: /forum/viewonline.php
Disallow: /forum/faq.php
Disallow: /forum/updates-topic
Disallow: /forum/stop-updates-topic
Disallow: /forum/ptopic
Disallow: /forum/ntopic
Disallow: /post-
Disallow: forum/post-*.html$
Disallow: forum/updates-topic.html*$
Disallow: forum/stop-updates-topic.html*$
Disallow: forum/ptopic*.html$
Disallow: forum/ntopic*.html$


thank you
0 Replies
 
Craven de Kere
 
  1  
Reply Fri 5 Aug, 2005 04:29 pm
Read the thread, all the answers are in here. But simply put, no it's not right. And you can find the reasons if you read (I wrote the tutorial so that I wouldn't have to write it over and over...)
0 Replies
 
aldemolay
 
  1  
Reply Thu 22 Dec, 2005 10:23 pm
I am having problems with bots signing up for my forum. can someone help me out on what exactly i need to do in order to stop them from being able to register. i know very little about server side code so as much detail as possible would be helpful. i just recently started messing with the forum part of the website i took over.
0 Replies
 
stanleyb23
 
  1  
Reply Sun 9 Apr, 2006 04:37 am
Hi Craven,

I ve done really good since im using your experieces! Lately i changed a few things in robot.txt. My english is not superb and i think i misunderstood a few things. Can u pls have a quick look at it?
Im using your seo mod. My forum is in a subdirectory called "Forum"

this is my robot.txt, i ve got a feeling that spiders dont see my about.html pages..

Code:
User-agent: *
Disallow: /Forum/admin/
Disallow: /Forum/images/
Disallow: /Forum/includes/
Disallow: /Forum/language/
Disallow: /Forum/templates/
Disallow: /Forum/common.php
Disallow: /Forum/groupcp.php
Disallow: /Forum/memberlist.php
Disallow: /Forum/modcp.php
Disallow: /Forum/posting.php
Disallow: /Forum/profile.php
Disallow: /Forum/privmsg.php
Disallow: /Forum/viewonline.php
Disallow: /Forum/faq.php
Disallow: /Forum/updates-topic
Disallow: /Forum/stop-updates-topic
Disallow: /Forum/ptopic
Disallow: /Forum/ntopic
Disallow: /ads/
Disallow: /links/
Disallow: /zoeken/


Its in my rootdirectory

Thnx in advance for looking at it.
0 Replies
 
Craven de Kere
 
  1  
Reply Thu 20 Apr, 2006 11:27 pm
aldemolay wrote:
I am having problems with bots signing up for my forum. can someone help me out on what exactly i need to do in order to stop them from being able to register. i know very little about server side code so as much detail as possible would be helpful. i just recently started messing with the forum part of the website i took over.


Enable CAPTCHA "Enable Visual Confirmation" in your admin panel (on the configuration page).
0 Replies
 
solobrian
 
  1  
Reply Tue 30 May, 2006 10:10 pm
How do I set up the robots.txt if my forum is in a subdomain called "forum" ?


ie: forum.mysite.com
0 Replies
 
Craven de Kere
 
  1  
Reply Wed 31 May, 2006 02:18 am
solobrian wrote:
How do I set up the robots.txt if my forum is in a subdomain called "forum" ?


ie: forum.mysite.com


Just as you would any other domain: put it in the root of the domain.

spiders should look for subdomain.domain.tld/robots.txt

e.g.

www.domain.com/robots.txt

or

subdomain.domain.com/robots.txt

As to what to put into it use paths relative to the robots.txt file.
0 Replies
 
solobrian
 
  1  
Reply Wed 31 May, 2006 01:55 pm
lets say the robot.txt is in forum.mysite.com/


The command inside robots.txt should look like this?:

Disallow: forum.mysite.com/admin/
Disallow: forum.mysite.com/images/
Disallow: forum.mysite.com/includes/

etc
etc
0 Replies
 
Craven de Kere
 
  1  
Reply Thu 1 Jun, 2006 12:19 am
No, it should be relative (please lookup "relative uri") to the location of the robots.txt file (which should be in root).

e.g.

/admin/
/images/
0 Replies
 
 

Related Topics

 
Copyright © 2019 MadLab, LLC :: Terms of Service :: Privacy Policy :: Page generated in 0.03 seconds on 10/19/2019 at 03:41:57