Help - Search - Members - Calendar
Full Version: server20 was down
Hostony Board > General Support > General Support
MartinB
server20 was down.
I need inform my clients about possible more future problems, or the situation is currently controlled ?
greetings.
Lehrer
Martin, the matter is that we have restarted the server because of an enormous load average on it. The customers responsible for that have got now their accounts suspended. We apologize for the inconveniences.
MartinB
Ok thank you Lehrer, no prob.
I hope that these users will be removed from the server...if the overload was caused for any activity who attemp to the AUP.
MartinB
I see various yellow and red alerts in the status page of cpanel.

CODE
Disk sda3 (/)    88 %    
Disk sda1 (/boot)    16 %    
Disk sdb1 (/home)    91 %    
Disk hdc1 (/home2)    84 %


I think it's not critical but i will let you know before it be problem.

(and you know, is not nice to see a lot of RED blinking alerts in a status page.)
brikface
QUOTE(Lehrer @ Jul 15 2005, 05:07 AM)
Martin, the matter is that we have restarted the server because of an enormous load average on it. The customers responsible for that have got now their accounts suspended. We apologize for the inconveniences.
*


Could you provide a little more info? In a separate thread I asked about the high load on server20, and if I hadn't stumbled into this thread I would never have known this was the problem.

Please answer these questions:

1) Are the accounts causing the high load now permanently off of server20?

2) Above what load figure for these machines do you start to troubleshoot? >1? >2?

3) If we account owners detect unacceptably high load (as determined by your answer to Question 2), should we contact you right away?

--
BF
brikface
Holy Mother of... Here's server20 uptime output from a minute ago:

18:48:13 up 1 day, 17:09, 1 user, load average: 17.62, 13.30, 9.31

Why? Explain.

--
BF
Lehrer
QUOTE(brikface @ Jul 16 2005, 06:51 PM)
Holy Mother of... Here's server20 uptime output from a minute ago:

18:48:13  up 1 day, 17:09,  1 user,  load average: 17.62, 13.30, 9.31

Why? Explain.

--
BF
*



Dear customers, the current status of the server is: 3 users, load average: 2.81
There are some temporary problems on that server, among which are tries to hack several users on the server, which were discovered in proper time.
By the way, such a high load average level(1 user, load average: 17.62, 13.30, 9.31
) can be explained as follows: we usually run our weekly backups on weekends, that is why the load average is high enough, though you could see only one user on the server.
Lehrer
QUOTE(mblendinger @ Jul 15 2005, 08:40 PM)
I see various yellow and red alerts in the status page of cpanel.

CODE
Disk sda3 (/)    88 %    
Disk sda1 (/boot)    16 %    
Disk sdb1 (/home)    91 %    
Disk hdc1 (/home2)    84 %


I think it's not critical but i will let you know before it be problem.

(and you know, is not nice to see a lot of RED blinking alerts in a status page.)
*



Martin, it's nothing serious, you don't have to panic.
Lehrer
QUOTE
2) Above what load figure for these machines do you start to troubleshoot? >1? >2?

3) If we account owners detect unacceptably high load (as determined by your answer to Question 2), should we contact you right away?

--
BF


brikface,

01-7.00 - Can be treated as normal load average level
07 - 20 - Acceptible,
Above 20 - is bad
Above 50 - you need to call a doctor smile.gif

If you see that the load average level exceedes 20, you should notify us (in case we are not working on it already)
brikface
QUOTE(Lehrer @ Jul 17 2005, 03:55 PM)
brikface,

01-7.00  - Can be treated as normal load average level
07 - 20  - Acceptible,
Above 20 - is bad
Above 50 - you need to call a doctor smile.gif

If you see that the load average level exceedes 20, you should notify us (in case we are not working on it already)
*

Lehrer, forgive me if I'm being argumentative, but I don't think a load of 20 on any *nix machine anywhere can be called acceptable. No way, man. LOAD is not an absolute metric and acceptable numbers can vary from machine to machine. In most cases any load greater than 1 indicates a sub-optimal condition. Greater than 2 and you'd better look into it as soon as you finish lunch. Greater than 3 and it's an emergency. As I say, different machines with different specs and configurations can vary, but they don't vary so much that a load of 7 is acceptable, nevermind 20!

server20 has some real problems. It's not uncommon for me to ssh in and literally wait for a full minute before the server digests my password and gives me a prompt. You say there is hacker activity. I hope that's it, rather than overcrowding...

--
BF
Lehrer
With my all respect, I can't agree that "Greater than 3 and it's an emergency".

The server is not overcrowded, as we are not setting up new users there for more than 1 year. You have seen for yourself: 1 user and load 17. Was the server up and running? Yes, it was. Nobody except one humble user was on it at that time. As I told you the current problem (that scared you) was in weekly backups.

But to cut the long story short, we are now scrutinizing the server and looking for further options.

Hope for your understanding, Lehrer
brikface
QUOTE(Lehrer @ Jul 17 2005, 05:21 PM)
With my all respect, I can't agree that "Greater than 3 and it's an emergency".
*

I said there is some variation, so if a machine is really packed to the gills with multiple CPUs, boatloads of RAM, fast disks and disk controllers, etc.,. then >3 may not be an emergency. Maybe >5 would be the must-deal-with-it-now point. But 7-20 acceptable? No. Never. Not for NASA's or IBM's fastest supercomputer and not for anything owned or leased by Hostony.

Yes backups are intensive, but if they are pushing Load beyond 3 (or 5 on a stacked machine) there is something very basically wrong. I've read that Cpanel's built-in backup is very problematic. Many providers have turned to 3rd party backup utilities or to homegrown scripts.

Anyway, I'm taking the time to make these comments because I just came through a nightmare with another service provider that I used for my commercial work (my Hostony shared-hosting is just personal stuff). Their problems began exactly like this. Initially their servers were humming along and everybody was happy. Then around October `04 we started seeing some Load spiking. We brought it to their attention in the forums. They disputed our assertions about what constitutes acceptable Load. The situation steadily deteriorated and we were seeing >8 Load for 8 of every 24 hours (usually midnight to 8am U.S. Eastern). They said "don't worry that's when we do backups". To make the story short, server performance declined so much that every single customer on every plan was threatening to leave. Then the company literally self-destructed and its remnants were sold to a group of Russian semi-gangsters in New Jersey.

Don't let this be the Hostony story. Work like hell to keep Load average within decent limits (in most cases less-than-3, in some cases less-than-5).

--
BF
Lehrer
QUOTE
Then the company literally self-destructed and its remnants were sold to a group of Russian semi-gangsters in New Jersey.


Sounds not optimistic at all. But believe me, this is not the HOSTONY story.
MartinB
laugh.gif laugh.gif laugh.gif laugh.gif laugh.gif
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2024 Invision Power Services, Inc.
IPS Driver Error

IPS Driver Error

There appears to be an error with the database.
You can try to refresh the page by clicking here