phpBB robots.txt tutorial

It's for the mod_rewrite portion of the mod.

0 Replies

LocanT wrote:

Craven or anyone else

Quote:
Disallow: forums/updates-topic.html*$
Disallow: forums/stop-updates-topic.html*$
Disallow: forums/ptopic*.html$
Disallow: forums/ntopic*.html$

are these file specific?

In other words if my forum is named myforumsshould they be in my robots.text

Disallow: myforums/updates-topic.html*$
Disallow: myforums/stop-updates-topic.html*$
Disallow: myforums/ptopic*.html$
Disallow: myforums/ntopic*.html$

Thank you very much in advance

is it not more correct this ? (look at / before myforums)

Disallow: /myforums/updates-topic.html*$
Disallow: /myforums/stop-updates-topic.html*$
Disallow: /myforums/ptopic*.html$
Disallow: /myforums/ntopic*.html$

0 Replies

yes

0 Replies

do you think is better to use for example

Disallow: /myforum/search.php

or (it's better to use ...) this

Disallow: /myforum/search.php*

Thank you
Graziano

0 Replies

wildcards are not standard.

0 Replies

I asked it because you used them on

Disallow: /myforums/ntopic*.html$

0 Replies

At one time I thought it would be adopted, but it hasn't picked up.

regex expressions would be very useful in robots.txt but adoption of them as a standard doesn't seem imminent.

In this thread I have given examples of how to work around this limitation.

0 Replies

Hi everyone,

I was wondering if anyone knows why my phpBB forum has been
suddenly dropped form the MSN index.

Edit (Moderator): Link removed

Could it be my robots.txt? What about my mod_rewrite?
Could my keyword url html pages have been red-flagged?

0 Replies

Do I have this right for my forums:

User-agent: *
Disallow: /forums/admin/
Disallow: /forums/db/
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/language/
Disallow: /forums/templates/
Disallow: /forums/common.php
Disallow: /forums/config.php
Disallow: /forums/groupcp.php
Disallow: /forums/memberlist.php
Disallow: /forums/posting.php
Disallow: /forums/profile.php
Disallow: /forums/printview.php
Disallow: /forums/privmsg.php
Disallow: /forums/viewonline.php
Disallow: /forums/faq.php
Disallow: /forums/ptopic
Disallow: /forums/ntopic

Looks like google is still indexing mysite.com/forums/post.html and topic.html

and not the forums/about.html links?

Thanks for the help

0 Replies

You don't have any rules to prevent indexing posts in your above post, however you do have it now on your robots.txt.

So if that was a recent addition expect it to take effect shortly.

If you want to hasten the process you can use search engines' removal tools.

For example, Google's is here:

http://services.google.com:8882/urlconsole/controller?cmd=reload&lastcmd=login

More information can be found here:

http://www.google.com/remove.html

0 Replies

Thanks I did recenlty add it after re-reading this post...

I'll keep you posted...

Mike

0 Replies

hello ,i use this robot.txt. i wonder everything ok or not?

Code:

User-agent: *

Disallow: /forum/admin/

Disallow: /forum/images/

Disallow: /forum/includes/

Disallow: /forum/language/

Disallow: /forum/templates/

Disallow: /forum/common.php

Disallow: /forum/groupcp.php

Disallow: /forum/memberlist.php

Disallow: /forum/modcp.php

Disallow: /forum/posting.php

Disallow: /forum/profile.php

Disallow: /forum/privmsg.php

Disallow: /forum/viewonline.php

Disallow: /forum/faq.php

Disallow: /forum/updates-topic

Disallow: /forum/stop-updates-topic

Disallow: /forum/ptopic

Disallow: /forum/ntopic

Disallow: /post-

Disallow: forum/post-*.html$ 

Disallow: forum/updates-topic.html*$ 

Disallow: forum/stop-updates-topic.html*$ 

Disallow: forum/ptopic*.html$ 

Disallow: forum/ntopic*.html$

thank you

0 Replies

Read the thread, all the answers are in here. But simply put, no it's not right. And you can find the reasons if you read (I wrote the tutorial so that I wouldn't have to write it over and over...)

0 Replies

I am having problems with bots signing up for my forum. can someone help me out on what exactly i need to do in order to stop them from being able to register. i know very little about server side code so as much detail as possible would be helpful. i just recently started messing with the forum part of the website i took over.

0 Replies

Hi Craven,

I ve done really good since im using your experieces! Lately i changed a few things in robot.txt. My english is not superb and i think i misunderstood a few things. Can u pls have a quick look at it?
Im using your seo mod. My forum is in a subdirectory called "Forum"

this is my robot.txt, i ve got a feeling that spiders dont see my about.html pages..

Code:



User-agent: * 

Disallow: /Forum/admin/ 

Disallow: /Forum/images/ 

Disallow: /Forum/includes/ 

Disallow: /Forum/language/ 

Disallow: /Forum/templates/ 

Disallow: /Forum/common.php 

Disallow: /Forum/groupcp.php 

Disallow: /Forum/memberlist.php 

Disallow: /Forum/modcp.php 

Disallow: /Forum/posting.php 

Disallow: /Forum/profile.php 

Disallow: /Forum/privmsg.php 

Disallow: /Forum/viewonline.php 

Disallow: /Forum/faq.php 

Disallow: /Forum/updates-topic 

Disallow: /Forum/stop-updates-topic 

Disallow: /Forum/ptopic 

Disallow: /Forum/ntopic 

Disallow: /ads/

Disallow: /links/

Disallow: /zoeken/

Its in my rootdirectory

Thnx in advance for looking at it.

0 Replies

aldemolay wrote:

I am having problems with bots signing up for my forum. can someone help me out on what exactly i need to do in order to stop them from being able to register. i know very little about server side code so as much detail as possible would be helpful. i just recently started messing with the forum part of the website i took over.

Enable CAPTCHA "Enable Visual Confirmation" in your admin panel (on the configuration page).

0 Replies

How do I set up the robots.txt if my forum is in a subdomain called "forum" ?

ie: forum.mysite.com

0 Replies

solobrian wrote:

How do I set up the robots.txt if my forum is in a subdomain called "forum" ?

ie: forum.mysite.com

Just as you would any other domain: put it in the root of the domain.

spiders should look for subdomain.domain.tld/robots.txt

e.g.

www.domain.com/robots.txt

or

subdomain.domain.com/robots.txt

As to what to put into it use paths relative to the robots.txt file.

0 Replies

lets say the robot.txt is in forum.mysite.com/

The command inside robots.txt should look like this?:

Disallow: forum.mysite.com/admin/
Disallow: forum.mysite.com/images/
Disallow: forum.mysite.com/includes/

etc
etc

0 Replies

No, it should be relative (please lookup "relative uri") to the location of the robots.txt file (which should be in root).

e.g.

/admin/
/images/

0 Replies

phpBB robots.txt tutorial

Related Topics

Quick Links

My Account

able2know