phpBB robots.txt tutorial

I don't understand.. why you don't want your topics and posts to be indexed by google ??

Yea, I don't get it anymore, at first I need to use * (wildcards)
A few pages later I don't (And from reading your posts I know why


So here is my Robots.txt

Disallow: forums/post-
Disallow: /updates-topic.html
Disallow: /stop-updates-topic.html
Disallow: /ptopic
Disallow: /ntopic
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /templates/
Disallow: /common.php
Disallow: /groupcp.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /profile.php
Disallow: /privmsg.php
Disallow: /viewonline.php
Disallow: /faq.php
Disallow: /updates-topic
Disallow: /stop-updates-topic
Disallow: /ptopic
Disallow: /ntopic
Disallow: /search.php

I have installed your SEO mod and the phpBB static URLs mod _rewrite 1.0.0 mod
(btw. from the default install I was missing extension.inc but I fixed that)

So is this a good robots.txt for an phpbbforum which runs Adsense!?

I know disallowing /search.php is a good one, because adsense is never going to give relevant results on that page, but the others?

(/faq, /admin, /images and /profile are correct to block as far as I know )

Can someone compare it to his or hers own robots.txt?

I like to know if I am correct about mine.

still waiting on an response ....

First of all, what is the entire point of blocking the robots from indexing these pages? Wouldn't it be most beneficial if we let them index the most amount of pages possible?

Second, how does this robots.txt file look:


And for an off topic question, do I put the .htaccess file, for the SEO Optimization Mod, in my forum directory or in the root of my website itself?

Thanks for being so helpful!

-Chris

bump

ha ha ... and an other bump ...

Since this seems to be a nice helpful forum I thought I would let you in a little tidbit for preventing bot regs and posts which will work without a bunch of visual confirmation junk. All you need to do is add the following lines to your apache conf file :

Edit [Moderator]: Link removed

The first two lines just ensure you have the required modules loaded, likely already done but good to double check. It should be obvious that ALL spam bots use the HTTP 1.0 protocol rather than HTTP 1.1. This will not prevent an actual user from registering and posting spam, as a normal browsers access will be via HTTP 1.1, but that is far less prelevant than the onslaught of spam bot activity we seem to be subjected to now.

The fact is this is quite elementary, and if you did not know how to configure apache for something this simple you likely should not be running an apache server at this time, since there are far more dangerous exploits that need to be addressed in order to have anything close to a secured server. You should research how to configure apache for hardened security and implement the proper methods before proceeding to run your server.

If you would like to prevent spam posts from actual users you can do that too, but it is much more involved and may require the AI monitoring software we develop for some servers.

Note: Since the moderators remove important links to text files, and since the filters here do not allow some very important parts of the conf file addtitions, you are going to have to go without my help. These forum are really nothing but a paranoia circus, my advice : learn how to run a website, how to manage server, and how to control a forum without making it unusable.

All is so good, but i don't understand this:

Disallow: /post-*.html$

Why i will disallow post-*.html from spidering ?
What is the point of this ?

We all want google to index our topics and post, didn't we ?

The post urls are redundant to the topic urls. By preventing the spiders from indexing 10 additional urls per topic page (or more depending on your pagination settings) they will spider more unique content.

Okey, i have now new question.

Here is the robots.txt:


when spiders craw the pages with "www." infront the urls, will they index the pages ?

there are two variants:

/forum/post....
and
forum/post... but which one is the best ?