Help - Search - Members - Calendar
Full Version: Error 403 when crawlers attempt to access robots.txt
Hostony Board > General Support > General Support
kazzy
Does anyone know why the server sends a 403 error instead of sending my robots.txt file??? Is this something that was pre-configured by the hostony admins or is there a workaround available?

Thanks.
MarkM
Are YOU able to access the file in your web browser?

How you tried CHMOD'ing it to 777?
kazzy
No I'm not able to access it from my browser. I also get a 403.

Yes I the file permissions are set to 777.
MarkM
I have no idea what it could be, honestly.

Try filling out a ticket.
Dima
I advise you to open a trouble ticket and provide us with your domain name so that we could investigate the issue more deeply.

usually this error is shown when there permissions problems,but there can be any other reasons.

Here are some pieces of advise on resolving this issue:

Resolving 403 errors - general

You first need to confirm if you have encountered a "No directory browsing" problem. You can see this if the URL ends in a slash '/' rather than the name of a specific Web page (e.g. .htm or .html). If this is your problem, then you have no option but to access individual Web pages for that Web site directly.

It is possible that there should be some content in the directory, but there is none there yet. For example if your ISP offers a 'Home Page' then you need to provide some content - usually HTML files - for the Home Page directory that your ISP assigns to you. Until the content is there, anyone trying to access your Home Page could encounter a 403 error. The solution is to upload the missing content - directly yourself or by providing it to your ISP. Once the content is in the directory, it also needs to be authorised for public access via the Internet. Your ISP should do this as a matter of course - if they do not, then they have missed a no-brainer step.

If your entire Web site is actually secured in some way (is not open at all to casual Internet users), then an 401 - Not authorized message could be expected. It is possible, but unlikely, that your Web server issues an 403 message instead.

Some Web servers may also issue an 403 error if they at one time hosted your site, but now no longer do so and can not or will not provide a redirection to a new URL. In this case it is not unusual for the 403 error to be returned instead of a more helpful error. So if you have recently changed any aspect of your Web site setup (e.g. switched ISPs), then a 403 message is a possibility. Obviously this message should disappear in time - typically within a week or two - as the Internet catches up with whatever change you have made.

If you think that the Web URL *should* be accessible to all and sundry on the Internet and you have not recently changed anything fundamental in your Web site setup, then an 403 message indicates a deeper problem. The first thing you can do is check your URL via a Web browser. This browser should be running on a computer to which you have never previously identified yourself in any way, and you should avoid authentication (passwords etc.) that you have used previously. Ideally all this should be done over a completely different Internet connection to any you have used before (e.g. a different ISP dial-up connection). In short, you are trying to get the same behaviour a total stranger would get if they surfed the Internet to your Web page URL.

If this type of browser check indicates no authority problems, then it is possible that your Web server (or surrounding systems) have been configured to disallow certain patterns of HTTP traffic. In other words, HTTP communication from a well-known Web browser is allowed, but automated communication from other systems is rejected with an 403 error code. This is unusual, but may indicate a very defensive security policy around your Web server.

Resolving 403 errors - CheckUpDown

The first question is whether the Web page for your URL is freely available to everyone on the Internet. If this is not the case, then you may need to provide two items 2. Web Site User ID and 3. Web Site Password for your CheckUpDown account - but only if your site uses HTTP Basic Authentication. The Web Master or other IT support people at your site will know what security and authentication is used.

If however your Web page is open to all comers and there have been no fundamental changes recently to how your Web site is hosted and accessed, then an 403 message should only appear if your Web server objects to some aspect of the access we are trying to get to your Web site. Because it indicates a fundamental authority problem, we can only resolve this by negotiation with the personnel responsible for security on and around your Web site. These discussions unfortunately may take some time, but can often be amicably resolved. You can assist by endorsing our service to your security personnel. Please contact us (email preferred) if you see persistent 403 errors, so that we can agree the best way to resolve them.


This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2024 Invision Power Services, Inc.
IPS Driver Error

IPS Driver Error

There appears to be an error with the database.
You can try to refresh the page by clicking here