This needs repeating: there is no way for a VPS provider not to have access to the internals of your VM; it is unrealistic to have such an expectation.
It might not be unrealistic to expect them to ask before looking, but if your concern is "one bad apple" then you've already lost.
I don't particularly care if they have access to the internals of my VM; I don't even care if they have the ability to log in to root using a specific ssh key (or a similar backdoor, so long as it's only accessible from within their network). Those things may be required for the reasons noted by the GoDaddy CSO.
I do care if someone uses my password to log in. I'm better than most folks at password security, but there's still too much opportunity provided by letting them have plaintext access to the password (I don't care if it's a decryptable format).
the important part is the difference between a compromised authentication secret and having local control over hardware/software. the password is now out in the open. there is now one more way for someone to root the slice other than getting control over the node(s) running the slice.
If they have root on your VM, they have your root password too. We're talking about maybe a couple hundred lines of code, tops.
It's clearly easier to get root passwords out of a database, but in the unlikely scenario where an internal employee sets out to sabotage the whole operation, a couple hundred lines of C code doesn't make the effort that much more unlikely.
Don't get me wrong; storing the passwords in the clear is very bad. The thing that is really going to go wrong? Someone's going to commit a change to their web app that coughs up everyone's password to an outsider.
I think there's a pretty huge difference in the risk of a bad apple writing code and inserting it into the system vs a bad apple looking up a password and logging in normally.
One requires significant sophistication and risk (of colleagues knowing your intentions are malicious), the other requires virtually no knowledge and low risk.
> If they have root on your VM, they have your root password too. We're talking about maybe a couple hundred lines of code, tops.
The default hash for crypt(3) (and thus /etc/shadow) in newer Linux distributions is salted SHA-512. Shouldn't that be significantly more difficult to crack than the old MD5 hashes? john didn't even support it when I checked a few months ago.
In effect, No. The computing power required to crack SHA-512 is infinitesimal compared to that for, say, bcrypt. However, modern crypt() implementations support multiple rounds just like bcrypt and that would make it significantly more difficult (time-wise) to crack. But then again SHA-512 is already being replaced due to some potential flaws found, so.... shrug
Kind of pointless to crack your password, though. I mean, it's not like you use the same one for more than one account.
No of course not. I'm sure we all use a different password for every box, webapp account, registrar, email address etc. I myself keep 100 or so passwords in my head, just like everyone else...
i know right? after the 16th character in the 96-character set it gets a little tiresome to type but i think keeping my twitter account secure is worth it.
I guess it's ok if they have local access. For cheap, crappy hosting it might be even ok if admins had access to passwords (but if they don't know about ssh keys, what can they fix really?). It is NEVER ok if some random support person can request your password because they need it, then write it on a piece of paper and log into your host as root once they get back home.
(what are you doing allowing root access via ssh anyways?)
If the filesystem is shared (i.e. server has / , and your vm1 is under /data/vm1 for example), they can access any files or databases without even logging in. If the filesystem is on a SAN they can access it read-only of course without you even knowing.
Thats true and it goes for dedicated servers and colo'd servers as well in some cases. The issue here isnt outrage about a host having access to your box, which almost all of them do anyways(IPMI, Root Disks, etc..) - but that they used his password to do it and without asking him.
I'm pondering a truecrypt'd system... unless they weed through the virtual-RAM to find the decrypting key, or they install a keylogger (likely against their end of the hosting agreement (though I haven't read any)), it'd essentially be secure against even them.
No, it wouldn't. The guest kernel has unencrypted access to the filesystem (it has to if you want to get anything in or out), and that's just a process running on their host. I'm reliably informed that getting the FS contents under those circumstances is relatively trivial.
Sadly, I can't say I'm really surprised, this is GoDaddy after all.
This is a company that has been shown to be consistently anti-customer time after time.
Their founder's latest blog post, on the GoDaddy homepage, is titled '5 things I wish I learned in Business School. Plus ... a smoking HOT blonde.' That pretty much sums up their cynical view of their customers as sheep to me.
This kind of freaks me out. If you told me somebody had cleaned out either my brokerage account or my GoDaddy account, I'd pray it was the brokerage account. There are much easier options for recovery from that.
It's not that I don't trust GoDaddy. They've been good to me. It is just that I don't trust every person who might happen to see their data base in the next decade.
Crikey, now I have to treat every password I've ever used there as compromised. Bleeeeeeech.
[Edit: It occurs to me that I probably lose consistency points for this since I've previously argued that plain text passwords are "not that big of a deal." Thomas, I owe you a drink.]
As 'regularfry ably pointed out, you gave up your root password when you opted for a VPS.
It's also standard operating procedure at hosting providers to escrow root access to systems. Some are better than others, but there are even colo providers that do this.
My advice on this issue is simple: no matter who your hosting provider is, your password should be random and system-specific. You shouldn't be using a Unix password so often that you need to remember it. You don't use GoDaddy itself enough to justify a memorable password either. If you lose a random password, who cares?
As 'regularfry ably pointed out, you gave up your root password when you opted for a VPS.
I don't have a VPS at GoDaddy. However, if their customer service reps can try his GoDaddy account passwords to attempt to log into his VPS, that means that GoDaddy is holding those in the clear somewhere, and that makes it very, very likely they are holding my passwords in the clear, too. That scares me because the prospect of a compromise of my GoDaddy account is really bad. It would allow an attacker to:
1) Compromise my DNS settings. Such as, e.g., MX records. So that they can compromise anything attached to my email address(es) without ever having to actually compromise my email provider. (Change MX record to point from Google Apps to L33t Hacker Krue, ask for password reset, watch as Internet obligingly delivers the email straight to their server.)
2) Compromise my SSL certificate.
3) My real nightmare: authorize transfer of my domains from my account at GoDaddy to their account at a disreputable registrar. They could either commercially exploit them or hold my business for ransom.
You don't think GoDaddy tech support can do that already, even without your password?
Remember, most of these companies have a somewhat-reasonably-tested customer visible web application, and an absolutely horrific "put-a-tick-in-the-URL-and-watch-the-fun" backend customer support web application.
Additionally, GoDaddy appears to have a pretty high turnover rate with all of their technical staff. I have yet to talk to someone who works there or has worked there who has had anything positive to say about the experience. Plain-text passwords and disgruntled staff strikes me as a potentially explosive combination.
You can tell I've been a SEO too long when the prospect of somebody yanking my ___domain name hurts a lot more than the prospect of somebody rooting me.
One thing people should be aware of is that almost no VPS hosting solution is secure.
The worst offender is Parallels/Virtuozzo(which I think godaddy uses).
On the linux side. A root user on the host node can simply run the command
vzctl enter 1111
and enter your vps without a password. To make things even scarier, when an admin enters your container this way. It doesnt leave a bash history file. You have a very small chance of ever knowing they entered the container at all.
Even if they dont want to enter your container. They still have full access to all of your files which are located in in the /vz folder on the host node.
Other vps solutions are slightly more secure. But it seems like %50 of the vps hosting industry is using virttozzo and most of them probably have no issue entering your container with or with out your permission.
It's simply not possible for a virtualised system to be secure from its host. Even if your filesystem is encrypted inside your slice, the kernel which decrypts it is an ordinary process running under their control. It's as well to assume that it can and will leak, and plan accordingly.
It's essentially DRM, except that you would be the media provider and the hosting company is the pirate. For your stuff to function you need to provide the cryptext, the decryption device and the key. So of course they have access.
Wow, you're letting them off the hook pretty easily don't you think? Seems like this is a major breach of trust, worthy of a little more sustained anger no?
Not off the hook, no. The issue still stands that they store the passwords in a retrievable format. About them accessing the servers, he said that they will change their policy to communicate first with the client, which is a big PLUS for me. If anyone wants their malware removal service, it is fine for me.
But just for calling back and talking about improvements, gives them a +1 (they are famous for ignoring their users).
Plus, that happened almost two months ago (see the date in the logs) and this post was on my draft for a while... So the sustained anger had time to pass.
Ah, didn't catch the date. I understand your anger may have subsided by now :)
But to me this still seems like letting GoDaddy off the hook. I understand they want to provide some sort of malware protection service for which they need to periodically log in to your vps with your password. The question is, if you are concerned about security at all, and you're not interested in taking advantage of that service, why would you continue to do business with a company that openly stores your password and makes it retrievable?
Seems like you shouldn't care if they have some sort of "procedure" in place for password retrieval. That's almost irrelevant. It's time to move your business elsewhere.
"I understand they want to provide some sort of malware protection service for which they need to periodically log in to your vps with your password."
This is so unusual, and so unexpected that it should have been in the TOS, at the top. This should NOT have been a surprise, with proper description of the REAL TOS, rather than just the written TOS. (I am assuming this isn't in the TOS, I'm not a GD customer.)
The encryption and process surrounding accessing passwords is meaningless. Who guards the encryption key, and how is this any better than storing passwords in the clear in some password-protected database?
The point is, the person with this key can view -- and possibly lose -- your password (which you may, but shouldn't, use in other places).
And once someone goes through this process, they can take it home with them.
The choice to do this, rather than to use an SSH key, is indefensible, and their explanation shows that they haven't got a clue.
They store all the passwords encrypted (not one-way hashed which is the recommended), and they can only be retrieved and reversed after a member of the security team opens a ticket and explains the reason for using the password (like to investigate malware)
They got root access (at least most hosts I know do), why would they need to reverse an encrypted password that would give them less access than what they already should have? Seems like BS.
Customer writes in, asks "I forgot my [FTP/mail/panel/etc] password, what is it?". Changing the password isn't necessarily an option, as the password may be saved in their FTP/mail client, or on their web designer's computer, or whatnot.
Thanks a lot for that suggestion/link. I'm looking for new dedi hosting and I really love those guys.
I'll be seriously thinking about moving to them. The most expensive combo package there is less than I pay right now. Plus, they're committed to privacy. Very very compelling.
I've used PRQ for about a year, and had no major problems. There was a DDoS attack at one time that broke things for a while, but other than that I've had no problems.
I had a dedicated server that ran without any problems, and when I had a few questions it was easy to get in touch via phone, mail and IRC.
This is a little different than just a registrar having access to your passwords. I don't have a server with them, so I must admit, I'm more concerned with whether or not they have my ___domain account password in cleartext. If they do, I'll switch registrars immediately, just on the principle of the thing.
That being said, I don't really care about them having your root password for your VPS. I really don't. Why? Because it's their server (physically). So, if they wanted access to it, they could just pull the hard drive and mount it in another computer. Your expectation of privacy just goes out the window if you don't own the hardware.
I agree with that (and I also confess that I didn't read it before signing up). However, it doesn't say that they will do that WITHOUT your permission (or even asking first).
They also store ftp and database user passwords in clear text. In fact, the used to show the ftp user passwords in their control panel. Now it's hidden, but you can still access ftp user passwords via their api and you can get MySQL user passwords by doing a "one-click install" of something and then opening the config file.
Correct. If having user service passwords stored in our database bothers you, log into a shell account and change the password with "passwd". We'll pick up the new password hash on our next user config and store the password internally as "changedbypasswd".
You can also view MySQL user passwords by clicking on the username in the panel (https://panel.dreamhost.com/index.cgi?tree=goodies.mysql). This shouldn't be a big deal, though, as you have to store the passwords in the clear in config files anyway.
We do not store control panel passwords "in the clear". They are stored as a hash and in a reversibly encrypted form; access to the plaintext passwords is heavily logged. We are considering removing it entirely -- the only reason we currently store panel passwords in any reversible format is because we frequently get emails from users who have forgotten their panel password and want us to email it to them.
Ah, Dreamhost. Where I got this support answer gem for a load average of 15+ on my shared server:
After looking at the ping and traceroutes, this is not an issue of server slowness, but of the connection slowness from DE to the United States, and as such there is nothing I can do to assist you in this matter.
Thats really scary, trust is always a concern when your in a managed hosting solution. How do we know that people from their company aren't taking advantage, even if the Godaddy isn't trying to.
A tool that I wish more people would use is PwdHash (https://www.pwdhash.com/). It's a system for generating psudo-random passwords for the web.
Basically, it will take some input from you (a password) and then hash that with the ___domain name of the site you're accessing and supply that when you actually submit the form. This means that my GoDaddy password would be "O0ErvdwEiy" instead of "asdfasdf".
It helps protect you against sites that may store your password in plain text because a compromise of one system does not mean a compromise of all systems.
It's fairly painless to use, but is only supported as a Firefox extension or through their website. There are some issues with sites that use different domains for their login servers or that use flash for their login dialog (yes, they exist).
Hostmonster.com also stores your password in plaintext. When I called customer support once they asked me for the last 4 characters of my password. So, I asked the support guy "Can you see just the last 4 or can you see my whole password?" He told me he could see my whole password. W. T. F.
I know that when I worked for a major hosting company in 2001, we had our own SSH keys added to every VPS. This gave us access without needing their password.
Right or wrong, it is very common at hosting companies.
Not really surprised by this. GoDaddy is horrible on so many levels, selling chained certificates without telling you what that implies until after you purchase; ganking peoples domains at the behest of third parties without going through UDRP.
And if you've ever had to deal with them you may agree with me when I say that their website constitutes criminal negligence on the part of the UI/UX designer. Deceptive and high-pressure sales tactics are a bad sign; especially in a company that wants so much of your trust.
The real question here is, who do you recommend as an alternative. Thankfully, the ___domain name registrar market is not monopolized by GoDaddy, but I have never used anybody else.
Worth repeating for the nth time: just because they have access to your password doesn't mean it's stored in clear. They could simply be using a reversible encryption algorithm.
This does not excuse them from accessing your VPS without your permission, of course, but the only thing you know about their password storage mechanism is it allows them to access your password when they need it. You have no idea what the procedure for this access is, or how exactly it's stored.
Worth repeating for the nth time: symmetric encryption algorithms for account passwords provide a misplaced sense of security and can be as risky as no encryption at all, due to changes in behavior and policy that result from misplaced expectations.
Reversible encryption algorithm that an admin can use to get my password and access my servers == same as clear-text to me...
The problem is not that they can get hacked and someone steal all their data, the problem is a malicious employee acessing my private server and my passwords. That's the issue.. unless they trust all their employees.
As others have pointed out, if you don't trust your host, you're hosed right from the start. If you want a private server, keep it in your closet.
Having the password reversibly encrypted means that if someone gets their hands on a dump of the db, they will at least not be able to automatically gain access to millions of accounts with no effort. Depending on the encryption scheme used, it may even be extremely secure - for example, decryption could require sending the encrypted string to a different, extremely secure server off the WAN that answers with the password.
In 15 years as a security professional and over 5 years directly consulting for big companies, little companies, locked down companies and lunatic companies, selling operating systems, browsers, parts of the power grid, cores of financial exchanges, retail banking applications, email management applications and to-do lists, I have never once seen the "extremely secure" system you allude to.
I have seen lots of "reversable encryption", though.
Maybe I've just gotten lucky in my career, and I just get the fun applications where people do this wrong. But there's no way GoDaddy did it right.
If I did this, that's how I'd do it. Have a separate computer, locked down to the max, except for a couple of functions: accepting HTTP requests POSTing an encrypted password, then sending back the decrypted string. That function would be severely rate-limited. Another function, to confirm the hash of passwords, would not be rate-limited, allowing high volumes of website access.
I could make that machine friggin' impregnable (and so could you). But yeah. No-one ever does it.
I wouldn't do it at all. If I wanted escrowed access to a VM, I'd stick a "break glass" SSH key on the box.
I'm not sure how "extremely secure" this "password-decrypting server" design really is, by the way. SQLI is often equivalent to remote code execution. Even when it isn't, XSS is equivalent to operator access, and operators can use the feature that decrypts the password.
Passwords are hazmat. You shouldn't be storing them, at all.
Yeah. I used to disagree with you, but now I agree.
You should write up a definitive guide for password security. For instance, I want to know if we should still use salts in the age of bcrypt, etc. Tell us what to do, man.
It's also worth noting that, because they control the storage for your hosted server, they can access it without your permission whether you want to allow them to or not. Just reset the ssh keys and bounce it ("oops, power failure!"), or just read out the data from your logs via a snapshot of the partitions, etc..
Still, storing recoverable passwords (which is what most people mean when they say "cleartext") without clear notice is a terrible idea for all the reasons people expect. The list can be stolen or misused, and this sort of thing happens routinely.
It's also worth noting that, because they control the storage for your hosted server, they can access it without your permission whether you want to allow them to or not. Just reset the ssh keys and bounce it ("oops, power failure!"), or just read out the data from your logs via a snapshot of the partitions, etc..
Definitely, which is why, as others have pointed out, you have to get a host you can trust.
Still, storing recoverable passwords (which is what most people mean when they say "cleartext") without clear notice is a terrible idea for all the reasons people expect. The list can be stolen or misused, and this sort of thing happens routinely.
Whilst I agree that storing cleartext passwords is not a great idea, I think the worse idea is to rely on every single web service you belong to to keep your authentication/authorisation credentials secure.
Your process of keeping your credentials secure should not depend on the security practices at any single one of the services you use.
People who publicly criticise hosts for storing passwords in clear, but then use the same password for multiple services, are nothing less than hypocrites. Remove the log from your own eye before picking at the straw in someone else's.
Not really worth repeating, because it's a relatively minor concern once the password is compromised. Someone else knowing a password == compromise. It's that they compromised user passwords without their knowing it that is the issue.
What you're saying is equivalent to "Yes, they stole your car, but they stored it in a safe garage". Not redeeming, at all.
No, this is equivalent to your gym using their master key to get into your private locker without telling you. They didn't steal the car - it's still there, right where it was.
Broken analogies really don't help the discussion.
>They could simply be using a reversible encryption algorithm.
I'm sorry, but this difference is completely trivial. Putting a password in some reversible encryption is just security by obscurity. In the worst case the people who managed to break into your site and steal all the passwords can just steal your magical decryption routine as well.
Not negative impact. I think his comment "dead man walking" was referring to the fact that they are less likely to innovate or make big changes to improve. There is always room for improve. Simple put, there are not the best VPS solution any more. But they are definitely one of the best. Others have caught up and in some cases surpassed slicehost, while Slicehost more or less stayed the same, they don't feel compelled to innovate or compete in terms of price and features. Rackspace dictates the terms.
I am a former Slicehost customer. It was simply put the best hosting solution I ever had. If I were looking for a VPS solution now, there are other good or even better solution out there.
GoDaddy's business practices are awful. Their view of the rights of ___domain owners alone was reason enough for us to move all of our domains to Gandi. More expensive, fewer features, but at least they respect their customers.
Who in their right mind would work with this funky company in the first place? They seem to be hiring morons. Or maybe they have the Superbowl models doing their Security.
It might not be unrealistic to expect them to ask before looking, but if your concern is "one bad apple" then you've already lost.