Help - Search - Members - Calendar
Full Version: Emergency maintenance
Hostony Board > General > News and Announcements
Stanly
One of our servers (65.254.35.18) was rebooted and didn't come back after reboot.
Our tech support team is already working at the server to locate and fix the problem. Some progress is already done but we having errors while booting.

I'll keep all of you updated at this problem.
Sorry for this accidental downtime.
bill
hi stanly, i dont mean to be nosy (my website is on 65.254.35.18) , but im curious as to the reason of the reboot in the first place seeing as to how it was in the middle of the day when the reboot occurred.

bill
TheColonel
how long until it's back up and running. my sites been down for almost 3 hours now.
Stanly
It seems we've RAID corrupted.
Tech is backuping all data from the server now and trying to fix RAID.
If it will be needed RAID will be rebuilded. Unfrotunately it may need some time.
No data will be lost.

Thank you all for your patience and understanding.
TheColonel
Any kind of time frame on this being fixed?
Serge
It looked like during cpanel automatic update system settings were corrupted. We are restoring the /etc directory of settings.

Also we are setting up a backup server DUAL XEON 2.4 - 2 Gb RAM 2x 73 Gb SCSI, so in case resolution of the problem on the current server will be very time consuming we could restore accounts from backups on teh new server.
drawstuf99
arg...my site is down on this server.

Where do you get your web host Hostony? Oviously not this one, as your site is always up!

Andrew dry.gif
bgp1
Would you rather have Hostony also go down? Then everyone would be in the dark. No email, no trouble tickets, not even a forum. Just be glad it isn't down. Then what would you do?
Serge
drawstuf99
To thell you the truth our main server was also touched with teh recent FreeBSd problems when files get corrupted. We had exim configuration messsed up but it is much easier problem to resolve since we did not have RAID.

On server 19 we have RAID5 in attempt to better protect users data, but so far we had more problems with it than with normal server setups.

We try to keep our main services up because all our customers depend on them and once main server is down they are blind. Actually we even think to distribute forums and helpdesk across different datacenters on different boxes to allow our customers to be able to communicate with us even in the worst case.

Now we are working on both servers: try to restore old one and also working on the new server restoring users data from backup.

New server is RedHAr Enterprise Linux 3.0 and this should fix ftp addon problem free bsd cpanel isntalls suffer from.
Nihilist
unfortunately, hardware particulary drives have issues, its inevitable. thanks guys,

any chance of throwing us a ETR at all ??
bill
the redhat linux seems more trouble free than freebsd servers i haven encountered.

have you guys ever considered running solaris ?
Serge
In fact freebsd is more stable and trouble free than linux. The real problem is Cpanel. I personally like FreeBSD more. Despite of announced support for FreeBSD and other systems Cpanel actually being developed for linux as primary system and all other OS releases lacks in support.

That's why for us is better to stick withg linux to create less problems to our customers with Cpanel features. On FreeBSD box there was a lot fixes that should be done manually after Cpanel upgrade. The fact is our linux customers experience less troubles with linux than with FreeBSD.
And this is a reason why we even won't consider running Solaris though it to some extend is better than FreeBSD and Linux.
Stanly
We're planning to finish the process of the moving in about 4-5 hours.
I'm still keeping you infomed.
Barrin!
Thanks for the info, Stanly. Good luck with the move.
Serge
Update.

34% percent data unpacked on teh new server. On the old server we were lucky to mount RAID, continue attempts to restore it completely or at least get more recent data backup. The one that we are restoring from was done on 7th March.
bill
serge, is there anyway to know if there will be data loss or atleast more recent backup?

i will have lost 90% of my data if the 'newest' backup is the 7th.
Stanly
Yes, it was at the 7th
bill
stanly, are there any nightly off site tape backup like alot of other companies do? and also what are the chances the data will be recovered off the raid5? b\c i did most of my changes after march 7th including a 200image gallery i was planning on releasing and DID not get a chance to save or backup. i have the originals but there was tons of enhancements only made to the server images. please tell me there is a way i can recover that information.
Stanly
The probability of recuperation data from raid is ~75%.
Now all of us are just are waiting while our tech is trying to fix RAID.
bill
ok stanly, even after the new machine is booted up, please keep me updated on the data of the raid because that will set me back alot of time that i cant afford.
Henry
Hi, I understand that as the customer it's my own responsibility to make sure I back up my site data regularly. However, in your published terms you state that "We back-up our servers on a daily basis" so may I ask why the backup you are using is a week old?

Unfortunately it's just not possible for me to be able to get to my cPanel to backup my databases daily, so I'm now likely to lose customer data unless the RAID recovery works. I had hoped that your daily backups would reduce the risk from my not being able to back up daily. So another question while I'm on the subject (I hope this isn't off-topic), can you recommend any automatic-backup scripts for mySQL databases? That's something that many of us affected might find useful.

Thanks for keeping us updated, it's very much appreciated.

Henry.
McMurtrey
I'm glad that Hostony is raiding it's servers?
Cause mirroring them would help everyone .
thanks.

Mirroring Servers Software

Naturally I just move 15 some domains here before this happen...
and I didn't back them up sad.gif

FTP Backup Software

McMurtrey
QUOTE
can you recommend any automatic-backup
scripts for mySQL databases? That's something that many
of us affected might find useful.


I found this on google, scrolll down to the 2nd reply by
serve-you.net Senior Member ....

I'll keep looking too....Backup MySQL - Cron Job Script
Alec
Nameserver ns19.hostony.net already points to a new server:
http://network-tools.com/default.asp?prog=...s19.hostony.net
So some sites are starting to work now

Also we are finishing backup process ans we'll keep you informed. We are backin' up data from Mar 7. Later we'll get a newer backup and restore it on your request.

Sorry for the inconveniences and thank you for your understanding
Nihilist
im still down sadly, but understandable, any estimated compeltion time ??
Serge
Sites can be accessed at 67.15.36.4

http://67.15.36.4/~username/

Conrol panel

Cpanel:
https://67.15.36.4:2083/

WHM:
https://67.15.36.4:2087/

Your old login/passwords should work.

In case your site is not visible, etc. please don't hesitate to submitt a trouble ticket to support and we'll checkit for you because all sites should have started to work about 3+hours ago.
Nihilist
Thanks for the update but a few questions

does that mean all the sites have new ips ??? if so we are going to have to wait for the new DNS to propogate or is the above solution only interim ??

also i am unable to login into any of the above with old user and pass
and what about ftp access would it be in the form of ??
ftp://67.15.36.4/~username/
Barrin!
For some unknown reason, I can't log onto my cPanel at the new IP you gave. It says my login information is incorrect. Do you know what's up?
Nihilist
can someone please explain how we are to get our sites back up ??
i can see it in the temp url, but my cpanel access doesnt work
Serge
During the last phase of the backup our reseller clicked sync FTP password and erased passwords. Now weare working on teh script that wil lrestore passwords from the backups.


Nihilist
Please submita trouble ticket with your site URL and we'll look into it.
Nihilist
thanks serge did that already , sorry i know all of you guys are probably still pretty busy

anyone elses control panel working ?? cant login
Henry
Any further updates? My site was working an hour ago and now it's gone again. cPanel wasn't working, but I guess that's still the password problem.

Henry.
Nihilist
were quickly approaching a 24 hour outage, i opened a ticket as you guys said a couple hours ago indicating i cant login into ftp and cpanel,

with all the data corruption i have no web site until i can access these things,
I have a ticket in queue waiting thanks guys
flamesrock
Is www.SimCitySphere.com on that server? I think it is, but I tried to access my home directory at http://67.15.36.4/simcitys/ and I couldn't.

-Thanks
Nihilist
you need to do this
http://67.15.36.4/~simcitys/
but it should be reachable on primary domain now
flamesrock
Thanks,

I still can't seem to reach at the primary http://www.simcitysphere.com though. Can you? sad.gif
Nihilist
i can get to my site through normal means (although there is no point since there is no web site there and cant upload one) I mean using just the web address, but not yours, i dont think the DNS for your name has been propagated yet.

I would open a ticket on your domain when i try to resolve it I dont get any ip info at all, old or new
flamesrock
Thanks wink.gif
zmenchhofer
any updates???? and when will the newer backup be restored??????????????

this is why i switched to Hostony because of the reviews but it is not looking good now mad.gif Thanks!
Serge
zmenchhofer

The server restore from 7 march was down about16 hours ago. If something still don't work for you please submitt a trouble ticket on support.hostony.com and let us know your domain and what doesn't work for you.

Thank you for your patience.
ZeroVoid
I feel sorry for you, my friend also lost a huge set of changes and client database of purchases and accounts, fortunatly for him he backs up daily by a cron job that sends changes and compressed backups to him or to his mirror!
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2024 Invision Power Services, Inc.
IPS Driver Error

IPS Driver Error

There appears to be an error with the database.
You can try to refresh the page by clicking here