Our server 64.74.112.74 is hacked and needs to be restored. Only apache is working on the
server now and we can't access it via SSH.

Unfortunately there is a tornado warning in the Atlanta and datacenter's support is not working now.
We are waiting until tornado will have gone to start reinstalling the server.

I reckon that server will be reinstalled within 6-12 hours.
Of course, we will do all possible to finish this task ASAP.

I am sorry for inconveniences.

This is unacceptable performance.
I have been down since YESTERDAY!!! (5-5-03)
And we have to wait ANOTHER 6-12 hours???? From what time?
Your timestamp shows that your message was posted today at 7:21 pm, but is that GMT or GMT+2 or what? Right now it's only 1530 Eastern Time where I am.

You're always telling your customers to NOT use the forum for critical questions. I also recommend YOU yourself NOT just use the forum for critical ANNOUNCEMENTS like this one. I alread re-opened my HelpDesk ticket on this issue BEFORE I looked in the forum. Now I find out what the explanation is.

You better get these problems resolved VERY quickly.

I think I can help with this part of the post "GMT time"
Although confusing at first, this is a way to get us all on the "same page" in a world with many time zones. The key is to set the time in your profile to your time zone, I live in New York so EST for me, which at the momment seems to work @ GMT-4 "daylight Savings screws it up I think"
Once set, you see all posts in your time!
If it says 3:30 on the post, it would mean "3:30 your time" no matter who the poster was. :wink:

As for the rest, I am worried as well, People count on me to keep there sites running and of the people whos sites I moniter it is my wife's site, and one of my test sites that is down right now, for a web administrator this can be "pofessionaly embarrasing". I will see what happens before I make judgements tho.

Before the Anouncement above, this was posted in another section of this forum



I take this to mean that they have a "couple hour job" ahead of them.

In the announcement above

I take this to mean the are having trouble getting a tech crew to the server.
If this proves to be untrue, or if they stop replying to our inquiries then they will have "show us their nature" and we should move on.
If they are telling the truth and they honor their responsibilities then this is a good chance to judge there service during an emergency.
If it is a one time problem, handled well, they may gain my trust.
If they act like some hosts who have cost me money instead, they will loose my business

I found a weather link for Atlanta

I ask this because when you moved to this new service two weeks ago you copied everything but did not copy the MX records on two of my domains. Thus I had a customer with 50 employees without email for 4 days until it was determined that was the problem.

I have two domins that have custom mx record settings...


www.kfg.com

www.advantravel.com

If these get screwed up again I will loose the clients (this assumes they are looking for a new provider right now)

Since I can't get email please copy reples to teamssc*nospam*@copper.net

take out *nospam* to send emails....

Because Hostony staff is in Russia (?correct or not?) it is middle of the night there and my guess is no one is working on this problem. When my MX records got screwed last week the last reply I got was 6:00pm (EST) and then tech support didn't started working with me again at 1:00am (EST).

So my fear is that the Hostony staff has gone to bed and they hope someone in Atlanta, where they outsource their hosting, will fix the problem...

I'm afraid there is no urgency here....

sscweb
Please be patient we are working on it and we are not in Russia. The server is restored and now we are restoring users data.

Shurik said:

<snip>
Unfortunately there is a tornado warning in the Atlanta and datacenter's support is not working now.
We are waiting until tornado will have gone to start reinstalling the server.
<snip>

C'mon, what's wrong with y'all thar in Atlanta? Worried about gettin' your 'doos mussed? We get a hurricane warning out here and we still go surfing, jeez! 8)

GOOD NEWS!!!!

It is now 2106 EDT.

All five of my sites are now showing an Apache page instead of "The page cannot be displayed"...


They're getting there!!!!!

Listen I'm supportive, I want you to succeed.... if you succeed I succeed... but we need communication here... and results...

I haven't seen 24 x 7 support.... and that was just last week when I was pretty screwed over on the MX record fiasco when the servers were moved....

And while I stand corrected, you are not in Russia you are in Ukraine? according to the owndership records of HOSTONY.COM.....


domain: HOSTONY.COM
owner-address: Serge Ryabchuck
owner-address: V.Porika 31/5
owner-address: 21021
owner-address: Vinnitsa
owner-address: Ukraine
admin-c: SR438-GANDI
tech-c: SR438-GANDI
bill-c: SR438-GANDI
nserver: ns7.gandi.net 80.67.173.197
nserver: custom2.gandi.net 80.67.173.18
reg_created: 2002-03-07 03:33:33
expires: 2004-03-07 03:33:33
created: 2002-03-07 09:33:35
changed: 2003-02-09 15:59:55

person: Serge Ryabchuck
nic-hdl: SR438-GANDI
address: V.Porika 31/5
address: 21021
address: Vinnitsa
address: Ukraine
phone: +38 0505384589
fax: +38 0505384589
e-mail: z555@z555.com

I wish that I had seen posts like this prior to subscribing. This host seems to continue to have a lot of problems (like getting hacked and lack of mail controls causing spam overload bottlenecks). If the apache page appeared at 7am est this morning, and content has not been reloaded by 9:30 pm on a <100 gig server, there sure is a problem. Its unacceptable.

Its also been close to 2 hours since I posted a helpdesk ticket, and not even the courteousy of a response informing me of the problem. Unacceptable also. If these sort of problems continue, I am going to need to ask for a refund and move my domains onward.

Added my account last Wednesday; Sunday it's down and I'm asked to upload my files again because they lost a server after an outgoing DOS attack My files total about 1 gig, transfering at about 476 which I got uploaded yesterday. So this morning I can't get into the control panel and they tell me it's due to a hacker. IF they take care of getting my files transfered, okay, one more change; IF I have to upload them again, let me recommend a VERY good host, www.sb3000.com. I only left them to get a higher GIG package, and after three years of service they only went down about twice for a few hours. Hostony beat that record in less than a week.

back at the ranch... martha and the kids were packing the wagon and trying to round of the farm animals.... pa had told them that there were greaner pastures in california.... so that's where they wer headed next... :arrow:

been hours since an update.... service should be restored by now...

No email for 14 hours... :evil:
No sites for 5 hours... :!:
No communication for hours... :?:

No hope for more then an hour... :cry:

Update:
By the moment server is restored and we also restored users files.
All configurations should remain the same.
The only thing is now ssh access now is set to jail in order to prevent hacking attempts.

In case you are experiencing any problems please feel free to post trouble ticket about it.

I'm back up!!!! Well atleast my webpage, the control panel is still down though

when should we expect email to be availible?

My Cpanel is now accessible. I am in the Middle East and it is now May 7th 8:10 am.

When I check server status it says email is"up" but it is DOWN.

When I log in to check email at any of my accts once I click the neomail logo the page comes up like this:

Software error:
Can't locate MD5.pm in @INC (@INC contains: /usr/lib/perl5/5.6.1/i686-linux /usr/lib/perl5/5.6.1 /usr/lib/perl5/site_perl/5.6.1/i686-linux /usr/lib/perl5/site_perl/5.6.1 /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl /var/cpanel/neomail /usr/local/cpanel/etc) at /usr/local/cpanel/base/neomail/neomail.pl line 34.
BEGIN failed--compilation aborted at /usr/local/cpanel/base/neomail/neomail.pl line 34.

For help, please send mail to this site's webmaster, giving this error message and the time and date of the error


When will email be working?

Thank you.

My control panel is back up. I'm assuming NeoMail and Horde will be back up soon. All is looking pretty good. Oh, and the jailshell, two thumbs up? Thanks for the good work guys!

sites are still not stable.... can't get to them 1/2 later

plus no email..... going to be an interesting morning....

I've been following their work fairly closely tonight. They did reboot the server about 15 to 20 minutes ago. But it came right back up. They are currently working on bringing email back up. I can see that the server is accepting the test email messages I send to it, so when I get up in the morning I should have plenty of email to delete

Travis, glad to know you are seeing progress.... I've not seen email yet on 8 domains... and 10 domains I've checked still can't be reached...

to sscweb:

I've just fixed your mail. It is working now. The problem was in the spamassassin.

Shurik,
I'm guessing the MD5 perl mod will be installed here shortly and NeoMail will be brought back up?

I've reinstalled MD5 and NeoMail is working now.
Going to fight with Horde

Thank you Shurik.

Hope you have a nicer day today than yesterday!

Keep well

All of my websites are now operational. I can also access cPanel.

HOWEVER, none of my users who use Horde for email can get in ... username or password rejected at the blue Horde screen. I have advised them to use Neomail which accepts them, but they're not happy ...

I would like to get them back to Horde. What's wrong with it NOW??

Please investigate and fix as soon as you can.

You mention spam assisian screwing with mail..... did you fix it on all 30 domains? I'm afraid to call my clients and ask them if they are getting email, or not...

I find a well informed customer is a lot more tolerable to downtime then an uninformed customer. Just call them, let them know your thinking about their best welfare and as long as they are not trying to use horde to get into there email, they are probably ok.

As for Horde being fixed, I just checked mine, and I'm also unable to log in so I just now notified the support group.

Ne created mailboxes work ok. We cannot test out your old horde logins since we don't have them. If you feel you have a problem with horde try just update your password for mail account. it should help since al the new created accounts are working.

In case you believe there is problem please notify support at support@hostony.com or via trouble tickets because you will be listing private information. Also trouble tickets are checked instantly when forums are just from time to time.

phpMyAdmin is not working in CPanel and I can access everyone else's web hosting packages through my WHM panel

Was fixed, completely down now

I just created a test account to check horde and I still get the "Login failed for some reason. Most likely your username or password was entered incorrectly." I'm sending the email to support right now.

i cannot login to my imap server with anything except for neomail - outlook express fails to authenticate even after i've used cpanel to reset my passwords.

now my imap server is completely unavailable:
randall@isber[~]% telnet redigital.org 143
Trying 64.74.112.74...
Connected to redigital.org.
Escape character is '^]'.

So final information on the issue.

We had a person who signed up with stolen credit card and used exploit to compromise a kernel.
It caused damaged to many services on the server but http functioned untill the momement we started restoring system on the new drive.
Because of the tornado warning techs were forced to leave datacenter for several hours. That's why we started to restore everything with a delay.

We had to restore OS and server configuration as it was before plust configure jailroot for shell access in order to limit potential hackers that try to attack system from inside the box.

After restore we were restarting services and tested users complaines about something being not functioning, etc. We had many false alarms because of mix +/@ and also there were some true problems.

As for now we have restored a system and helped customers that posted trouble tickets with complete details about their accounts.
I assume that people who posted above about email problems got answers through troubl tickets. If not feeel free to submit a trouble tickets with all details required for us to reproduce your problem.

Sorry for the caused troubles and thanks you for your patiemce.

Every hosting company has a hack or some other kind of meltdown now and then. We'd rather have 100% uptime of course, but that isn't reality. If meltdowns are common due to poor tech and maintenance, that's one thing. But for the occasional inevitable event, it's how they respond that's important. From what I have seen here, Hostony got on it as soon as possible did what they had to do, and those things took some time to complete. There were some complaints that information was not forthcoming - maybe Hostony could put a little more effort into updates and customer response next time. When my site on this server went down, I saw the initial notice with an estimate of repair time that, for my site at least, proved to be fairly accurate.

I say thanks to the staff for getting things back up in a timely manner even though those wusses in Atlanta were afraid of a little whirly-wind. Just don't let it happen too often. :wink:

Thank you for handling the problems as promptly as you could.
I was impressed that not a single bit of data was lost from my sites or my databases. There were many things you had to repair so I will continue to check everything for a while, but I feel confident that if I discover anything it will be handled if I submit properly to you.
I only had one report from someone using my e-mail system about Horde but I just checked my own mail account and was able to log in ok, I will get back to him and see if his problem is also solved, as I believe it prolly was. if not I'll let ya know

keep up the good work!

We must power off the server so that to get old HDD from the box. Because the server can be down for few minutes.

Aren't your server drives SCSI hot-swappable? They should be. In our datacenter here (where I work in the IS dept), we have 17 huge servers and an EMC SAN. Everything has redundancy galore: NIC's, internal power supplies, HDD, fans, Cisco switches and routers. We operate our business 24X7X365 for manufacturing and distribution sites around the world. We CANNOT go down. Period.

Now that that the fires have gone out, I want to compliment you on getting things back under control in a very reasonable timeframe. I suspect many of us the past few days have been frustrated because of past horrible experience at OTHER webhosts and we were thinking "Oh, no, not AGAIN!" I confess I had a few minutes of second thoughts about leaving my previous host. Unlike them, you did quite a good job of TRYING to keep everyone informed. Not easy when you're ALOS trying to fix things.

As for the tornadoes in Atlanta, the office building where we have offices was evacuated to the basement twice that day. Your techs did the right thing when they were told to leave. That's the LAW.

My suggestion: when such issues strike (and they will again) put a quick link right on the FRONT page of the Forum. It seems to me too many disconnected conversations were going on in many sub-topics that could have stayed in one place. I spent a lot of time digging around for answers.

Thanks again!

Update the maintanance for this box should be around 12 PM GMT. Estimated downtime should be arounf 5-10 minutes.

jchrastina

The other Xeon has SCSI but this box has IDE drivers. When we are talking about shared web hosting we don't tell about IBM iseries that start at 45К, etc and OSes that allow inmemory upgrades without need to reboot. Customers that want shared web hosting don't want to pay much otherwise they would pay for dedicated box that is closer match for increased security and stability because you are the only who has access to mailbox and use its resources, etc.

That's why common hardware used in shared web hosting industry is a Linux powered box. Since it should be rebooted to see many things like additional memory, CPU, etc. and all the data can be put on the half of the IDE 80 Gb drive - it is not wise to use hotswapable hardware that costs way more and increase bottom costs that it became wise and much cheaper for the customer to buy a low end dedicated box to get the same level of security. The other reason sinceit is not a big system where that major par of HDDs is used for storage it is also not wise to have hot swapabe systems because HDD here contains all the system and users data and even when there is 2 HDDs and one contain only users data there will be still considerable downtime because after the OS HDD swap we'll need to configure control panel and services to see users data. Just as we had to do last time.

THIS SERVER IS DEAD AGAIN

The server was overloaded by some script. We rebooted it and everyhting looks ok now.
Now we are investigating what caused the high load.