Help - Search - Members - Calendar
Full Version: What are search engine defaults?
Hostony Board > General > Anything and Everything
brikface
Heyo, I'm a new Hostony user, Can someone tell me what the defaults for search engine indexing of Hostony sites are? That is, if you do nothing with the "Submit to Search Engines" applet in Cpanel, do any search engines grab any of your pages?... Eventually I would like to expose some directories on my site to engines and hide others. Will that be doable? Thx, Brikface.
Vanya
Your site is indexable by search engines.
Hotlink protection(Cpanel - Hotlink protection) should help to hide some pages.
dragonfli
A typical way to discourage search bots from certain pages or folders while "allowing" them to spider others is a small file called robots.txt
QUOTE
Search engines will look in your root domain for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard.

Not all bots respect this, but the reputable ones do.

Here is a link to a quick tutorial about this.
http://www.searchengineworld.com/robots/robots_tutorial.htm

Also a google search for "robots.txt" will supply you with much information about this subject.
brikface
Thanks, Vanya and Dragon-- but I specifically wanted to know what happens if one does nothing with the cPanel applet or with robot.txt files: is the entire site indexed; is just the top level indexed; is nothing indexed?
dragonfli
If a bot hits your site and you don't give it rules (robots.txt), you can pretty much assume it will index everything it sees.
The submissions, done manually, or via cpanel are just invitations to come crawl and a "map"(url) to your site. Without the invitation they will eventually find you anyway, after all it's their job, and they are only dumb, untiring bots. If you want to see if they've been there just look in stats in your cpanel; I like to check for them using awstats, but that is just a preference.

I am no expert in SE, that's why I supplied the link, but they each have thier own rules of behavior (that change). Research is needed to discover what each bot may or may not do specifically when they find your site.

Some links that describe some of the SE spiders and how they behave (what they will index - will they come without submission):

Infoseek
Atavista
Lycos
Excite
Webcrawler
Google
Northern Light

and Vanya did answer your question regarding will they index without submission
QUOTE
Your site is indexable by search engines.
brikface
Thanks Dragonfli, that pretty much clears it up. Vanya's quote could be read a couple of ways-- he could have meant the site is "indexable" only after you take such-and-such a step. Stating things without ambiguity is probably one of the more challenging aspects of tech support... Anyway, I'm hoping I picked a winner with Hostony. One never knows which companies will be standing the same time next year, but something tells me Hostony will.
Danimal
Hey brikface,

Though spiders will find your site eventually, if you want to be listed on Search Engines now, the best way to do this is to either go to the ones you want (like Google or Yahoo) and submit your site, or use the "Submit to Search Engines" function. It will speed up the process drastically as they'll send out their spider robots right then, rather than whenever they get around to it.

Hope this helps.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2024 Invision Power Services, Inc.
IPS Driver Error

IPS Driver Error

There appears to be an error with the database.
You can try to refresh the page by clicking here