robot.txt 101 for dummies?

CDK or anyone else that can help....

Is there a tool online that can spider my site and tell me how well it "might" do against different search engines and possibly give suggestions on how to improve my site for SEO? RIght now I'm using bcentral's submit-It product to tell me what's good and what's considered bad, but according to what I've learned here, some of the suggestions that bcentral suggests that I do I probably shouldn't.

Thanks...

Tim

Yes there are tons, but they are as accurate as blind monkey throwing dartboards.



Lesson number 1 with SEO:

Automated tools are usually not worth anything and usually give horrid, outdated and patently incorrect advice.

Automated SEO tools have more to do with making money off of n00bs than anything else.

I seem to have some new vistiors from Google. The IP's I listed earlier in this thread you said were "fresh" bots. Well are these the "deep search bots"

IP address: 64.68.82.27
Host name: crawler10.googlebot.com
IP address: 64.68.82.181
Host name: crawler14.googlebot.com
IP address: 64.68.82.135
Host name: crawler13.googlebot.com
IP address: 64.68.82.136
Host name: crawler13.googlebot.com
IP address: 64.68.82.164
Host name: crawler14.googlebot.com
IP address: 64.68.82.10
Host name: crawler10.googlebot.com
IP address: 64.68.82.79
Host name: crawler12.googlebot.com
IP address: 64.68.82.159
Host name: crawler14.googlebot.com
IP address: 64.68.82.44
Host name: crawler11.googlebot.com

I sure hope they are, I've been waiting patiently to get "crawled" on...

Tim

hmmm....

I just read that the freshbot has the ips of 64.* and that the deepbot has the 216.* IP's I guess I still have to wait...this is the first time, I've seen this type of activity on the site in such numbers from google.

Well the good news is that Google has been phasing out deep crawl updates and they plan to rely on freshbot style updates.

They used to do a major update around once a month but now they do many more mini updates.

So the turn around is a lot faster these days and the days of waiting for deepbot may be over.

CDK,

I don't want to detract or redirect ppl from the great information that you provide here, but can you suggest a site that has some of this new info about google or some site that lists "current google" actions that you refer to. I sure would be interested in reading a little bit about the changes.

Tim

BTW, I've had these freshbots on my forum nearly all day. I've seen mulitple instances for over 6 hours now.

There are hundreds of thousands of such sites. Search engine watch by Danny sullivan is the leading Search engine industry news site.

But if you want the info, the info is only found on the search engines themselves.

Any list of tips and tricks will be outdated fast, if you are looking for my source of info it is the engines themselves.

The algo is guessed at through observation of their behavior.

hi all.

warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/a2k-post*.html$
24 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/a2k-view-poll*.html$
25 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/updates-topic.html*$
26 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/stop-updates-topic.html*$
27 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/ptopic*.html$
28 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/ntopic*.html$

url:
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

why warning ? pls help me.

hello .

Able2Know.com SEO 2.0.0 added my site. but not listed search engines my forum. ?

mod added date: 11 agu

It means exactly what it says, using wildcards in robots.txt is not a standard even though Google supports it.

Other spiders support other non-standard features as well.

I will answer this question on the SEO mod thread, and not in all three of the threads where you asked it.

thanks for your answer.

hello all.

-----
Robots.txt Validator

http status: 200 OK

Syntax check robots.txt on http://www.lavinya.net/robots.txt (219 bytes)
Line Severity Code
It validates, but has some bad style.

2 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/post-*.html$
3 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/updates-topic.html*$
4 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/stop-updates-topic.html*$
5 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/ptopic*.html$
6 warning Possible Missplaced Wildcard. Although Google supports wildcards in the Disallow field, it is nonstandard.

Disallow: /phpBB2/ntopic*.html$


robots.txt source code for http://www.lavinya.net/robots.txt
Line Code
1 User-agent: *
2 Disallow: /phpBB2/post-*.html$
3 Disallow: /phpBB2/updates-topic.html*$
4 Disallow: /phpBB2/stop-updates-topic.html*$
5 Disallow: /phpBB2/ptopic*.html$
6 Disallow: /phpBB2/ntopic*.html$
7 Disallow: /refer/

-------
why warning ? pls help me.

Read the warning, it tells you why repeatedly.