phpBB robots.txt tutorial

Forums: Phpbb, Web Development, Robots
Email this Topic • Print this Page

robots.txt is a file that must be placed in the domain's root directory. Search engine spiders will load it and then follow the rules (if they are "good" spammer spiders and other miscreant spiders will not follow the rules).

robots.txt is basically the best way to keep certain pages from being spidered and indexed by search engines.

The following is an example for phpBB.

Create a file called robots.txt

Then add the rules, preventing spiders from spidering the pages you do not want them to spider.

Here is an example for a phpBB install in a subdirectory called "forums". If you have it installed in a different directory you need to change this.

Code:

User-agent: * 

Disallow: /forums/admin/ 

Disallow: /forums/images/ 

Disallow: /forums/includes/ 

Disallow: /forums/language/ 

Disallow: /forums/templates/ 

Disallow: /forums/common.php 

Disallow: /forums/groupcp.php 

Disallow: /forums/memberlist.php 

Disallow: /forums/modcp.php 

Disallow: /forums/posting.php 

Disallow: /forums/profile.php 

Disallow: /forums/privmsg.php 

Disallow: /forums/viewonline.php

Disallow: /forums/faq.php

Disallow: /forums/updates-topic

Disallow: /forums/stop-updates-topic

Disallow: /forums/ptopic

Disallow: /forums/ntopic

Disallow: /post-

The last few pertain to the Able2Know.com SEO MOD. You might also consider preventing search.php from being spidered.

Topic Stats
Top Replies
Link to this Topic

Type: Discussion • Score: 2 • Views: 71,755 • Replies: 70

No top replies

hello craven,

what would be the benefit or the consequence to throwing in search.php in that robots.txt file?

thanks!

zm

0 Replies

Well,

This type of page:

http://www.able2know.com/forums/search.php?search_author=zoomsan

Can either help or hurt, it depends on your site's "link flow" and a couple of other trivialities.

Usernames can be SE queries and for that it can help (especially if you have thousands of members because the variety means a couple of obscure hits on SERPs.

But if you have very context targeted content in smaller numbers it might just be seen as dillution of your content by search engines. To tell which it is for your site you need to follow the search engines closely as well as your log files.

0 Replies

Sounds great!

I noticed that on the post here you left out:

Disallow: forums/post-*.html$

but you have that in the Mod file ....

what is the significance of this line and should it be included or left out in your opinion?

Thx

zm

0 Replies

I left it out because that's not vanilla phpBB.

The significance is that all the post URLs are just seen as duplicate content.

0 Replies

hello craven,

do you feel there is any reason not to Disallow login.php in the robots.txt file?

Thanks!

zm

BTW will that link mod be released in the other thread? Or perhaps a seperate thread? And do you use the link mod, as I noticed you disallow links completely ... Thanks in advance! And I will PM you a link so you can see your mod in action.

0 Replies

I see no reason not to disallow login.php on a vanilla install.

Yeah, the links mod is in effect here. See: www.cnn.com

0 Replies

Fantastic .. where should I look for the link mod once its done?

Also, what is the reason for not having the "/" in front of those files at the end of your robots.txt? I am sure it is something simple but it would be helpful to know.

Thx.

zm

0 Replies

Doh, that's a typo.

0 Replies

Craven or anyone else

Quote:

Disallow: forums/updates-topic.html*$
Disallow: forums/stop-updates-topic.html*$
Disallow: forums/ptopic*.html$
Disallow: forums/ntopic*.html$

are these file specific?

In other words if my forum is named myforumsshould they be in my robots.text

Disallow: myforums/updates-topic.html*$
Disallow: myforums/stop-updates-topic.html*$
Disallow: myforums/ptopic*.html$
Disallow: myforums/ntopic*.html$

Thank you very much in advance

0 Replies

If you are using a different directory you will need to change the "forums" part. Basically, you seem to have the right idea.

0 Replies

Craven thanks

On a similar subject

the mod_rewrite part

Quote:

OPEN ]------------------------------------------
#

.htaccess

#
#-----[ ADD ]------------------------------------------
#

RewriteEngine On
RewriteRule ^forums.* index.php [L,NC]
RewriteRule ^post-([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?p=$1&highlight=$2 [L,NC]
RewriteRule ^post-([0-9]*).* viewtopic.php?p=$1 [L,NC]
RewriteRule ^view-poll([0-9]*)-([0-9]*)-([a-zA-Z]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&vote=viewresult [L,NC]
RewriteRule ^about([0-9]*).html&highlight=([a-zA-Z0-9]*) viewtopic.php?t=$1&highlight=$2 [L,NC]
RewriteRule ^about([0-9]*).html&view=newest viewtopic.php?t=$1&view=newest [L,NC]
RewriteRule ^about([0-9]*)-([0-9]*)-([a-zA-Z]*)-([0-9]*).* viewtopic.php?t=$1&postdays=$2&postorder=$3&start=$4 [L,NC]
RewriteRule ^about([0-9]*)-([0-9]*).* viewtopic.php?t=$1&start=$2 [L,NC]
RewriteRule ^about([0-9]*).* viewtopic.php?t=$1 [L,NC]
RewriteRule ^about([0-9]*).html viewtopic.php?t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 [L,NC]
RewriteRule ^mark-forum([0-9]*).html* viewforum.php?f=$1&mark=topics [L,NC]
RewriteRule ^updates-topic([0-9]*).html* viewtopic.php?t=$1&watch=topic [L,NC]
RewriteRule ^stop-updates-topic([0-9]*).html* viewtopic.php?t=$1&unwatch=topic [L,NC]
RewriteRule ^forum-([0-9]*).html viewforum.php?f=$1 [L,NC]
RewriteRule ^forum-([0-9]*).* viewforum.php?f=$1 [L,NC]
RewriteRule ^topic-([0-9]*)-([0-9]*)-([0-9]*).* viewforum.php?f=$1&topicdays=$2&start=$3 [L,NC]
RewriteRule ^ptopic([0-9]*).* viewtopic.php?t=$1&view=previous [L,NC]
RewriteRule ^ntopic([0-9]*).* viewtopic.php?t=$1&view=next [L,NC]

#
#----

The firts line

Quote:

RewriteRule ^forums.*

Should this also be changed to (in my case) ^myforums.*

0 Replies

Nope, in fact that is pretty much useless. It's just there in case you want to link to index.php as forums.html

That being said, change it to whatever ya want, it won't be used unless you link to it that way.

0 Replies

LocanT wrote:

Craven or anyone else

Quote:
Disallow: forums/updates-topic.html*$
Disallow: forums/stop-updates-topic.html*$
Disallow: forums/ptopic*.html$
Disallow: forums/ntopic*.html$

are these file specific?

In other words if my forum is named myforumsshould they be in my robots.text

Disallow: myforums/updates-topic.html*$
Disallow: myforums/stop-updates-topic.html*$
Disallow: myforums/ptopic*.html$
Disallow: myforums/ntopic*.html$

Thank you very much in advance

One more Should these have a / before

exmaple

instead of
Disallow: myforums/updates-topic.html*$

it would be
Disallow: /myforums/updates-topic.html*$

my forum is not on a sub domain(perhaps I should have done) it is in a folder on the domain

Thanks

0 Replies

It should have the slash.

0 Replies

Google Adsense and Robots.txt
I have Google Adsense on each page of my forum. The Adsense Bot needs to visit each page to deliver relevant ads. Do you know if your example robots.txt file causes problems with Adsense? Thanks.

0 Replies

It shouldn't but you may want to allow post urls as the bot will want to monetize those as well.

0 Replies

For some reason, Google still indexes login.php and search.php. If anybody sees something wrong with my robots.txt, let me know. My phpbb files and robots.txt are in the root directory of my site. Thanks Smile

Code:

User-agent: *

Disallow: /post-*.html$

Disallow: /updates-topic.html*$

Disallow: /stop-updates-topic.html*$

Disallow: /ptopic*.html$

Disallow: /ntopic*.html$

Disallow: /admin/

Disallow: /db/

Disallow: /images/

Disallow: /includes/

Disallow: /language/

Disallow: /templates/

Disallow: /common.php

Disallow: /groupcp.php

Disallow: /memberlist.php

Disallow: /modcp.php

Disallow: /posting.php

Disallow: /profile.php

Disallow: /profile.php?mode=register

Disallow: /privmsg.php

Disallow: /viewonline.php

Disallow: /faq.php

Disallow: /search.php

Disallow: /search.php?search_id=unanswered

Disallow: /login.php

0 Replies

Adam,

Use this link, and have Google remove urls based on the robots.txt exclusions:

http://services.google.com/urlconsole/controller

See if that works.

0 Replies

First let me say: Craven, you're awesome! Thanks for all the effort you put into helping us out on these forums.

I tried what you suggested, and here's what Google said:

Quote:

URLs cannot have wild cards in them (e.g. "*"). The following line contains a wild card:
DISALLOW /post-*.html$

Similarly, when I check my robots.txt file with online programs, they pretty much all tell me the same thing - wildcards are a no-no.

Now, I don't actually have a problem with Google indexing ntopic or ptopic or any of the additions from your mod_rewrite text, but I'm wondering if there's some inadvertent correlation between that and the problem I'm having.

Something else I just realized - I have this tag inside overall_header.tpl:

Code:<meta name="robots" content="index,follow">

Could that be telling the search engines to index everything regardless of robots.txt?

Thanks Smile

0 Replies

phpBB robots.txt tutorial

Related Topics

Quick Links

My Account

able2know