Watch out for the bad Y-axis labels on this post: they go from min to max instead of 0 to max. This doesn't change the ranking, of course, but exaggerates the relative difference.
I think so. It's an easy-to-miss detail that really changes your perception of the data. Ideally the graph would indicate this itself by having a broken axis, but that's probably a pain to do.
I included a note at the top of the article. The graphs are from Google's API and were a pain to get in the state they are now. Not going to change them in this lifetime;-)
It also depends on whether the main purpose of the graph is to show changes along the ranged axis (as if from period to period over time), or the magnitude of differences between the lines on the ranged axis.
If the latter, a zero origin is fairer. While the day-to-day differences are of some interest here, the main interest is provider vs. provider, so the axes are not ideal. They make the provider-to-provider differences seem larger.
You're in much greater danger of having poor performance and downtime from your own code rather than anything to do with your host. So, before you jump ship from one to another based on benchmarks, think about how much time and effort such a move will take and ask yourself what you could spend it on.
If you're running a web app, my bet is you'll get a much better ROI on time if you pick an expensive DB query you're running and try to analyze/optimize it.
All that said, if you haven't yet picked a host, this does offer some interesting insights. Still, I would rank APIs, GUIs, management panels, and helpful support as way, way more important than raw performance.
I use prgmr, but it's only personal use right now. I paid the annual for the 1GB plan. The monthly cost is quite low and it gives me my own little server to deposit all my apps on and what not. I'm very happy with it. The performance is good enough for me.
I have had no issues at all. Highly satisfied customer here.
Did you have any setup problems? I tried Prgmr a few months ago and my instance wouldn't boot for some reason. Tried emailing the support but didn't get a response. It was only $6 to give it a shot, but I guess I'm not smart enough to use it.
really? did we at least get you a refund? we do sometimes drop stuff, but we try to at least refund everyone who we don't get going.
(the vast majority of or problems are provisioning-related, as well... we're working on that. But still. If you didn't get the refund email me and I'll see to it.)
No contact at all I'm afraid. Don't worry about the refund - I've been ignoring the emailed invoices ;-) What I'd really like is to give you another try, so I've sent you an email with the details.
I signed up, emailed Luke my SSH key and said CentOS. He emailed back saying it was ready and off I went to setup it up.
Very minor issue when I first signed up, but I hadn't even logged in to the box yet. I emailed Luke and in about an hour I got a response back saying it was fixed.
I'm not using it for anything critical, so if something goes down or seems not correct, I might email and ask what's up, but usually I probably won't even notice or care enough.
I also follow his twitter, which he seems to keep updated if something is not functioning as it should.
Prgmr have been quite accommodating in my experience. I'd have to characterise their support as like working with a peer. Using their support address I was able to provision a couple of instances on separate hardware with an additional floating IP and at a cost that borders on a quarter of what I'd have paid with Linode.
Not responding in time is the worst thing such a company can do. My first slicehost instance didn't start as well. It took 31 minutes from sending an email to getting a "problem solved" response. Stuff always happens, it's how they react that makes the difference.
Not necessarily. This article ignores the "neighbor" factor. That is, performance might be highly dependent on what the neighboring VPSes on the same compute node and storage node are doing. It could very well be that on Linode he was placed into a underloaded system that did not have other active VPSes on it or even have been filled out to capacity yet. It's not possible to tell. It may just reflect a better allocation system than the other providers, assigning compute nodes in a round robin fashion, not in order.
So, it's a little unfair to do a benchmark when 95% of the system's resources are taken up by an unknown factor.
(Note: I don't host with any of these guys, so I'm not defending Slicehost or anyone else. In fact, I run my own hosting company and I'm working on our own cloud offerings right now.)
You should put your hosting company website in your profile. Personally, when a user on HN says something intelligent, I like to check out their company.
One thing a lot of people have seen with Amazon is high instance to instance variation. Generally if one starts up a batch job of 10-100 odd instances all doing the same work, you'll see up to a 10x difference in throughput between best and worst instances.
I'd speculate amazon has many different generations of hardware deployed and the EC2 Compute Units do not really capture the differences between these, particularly when it comes to memory bandwidth.
As a matter of fact I ran the benchmarks on one instance in EU and one in US before I started a week long run on one US instance. I simply could not believe the numbers I was getting from what should be a larger share of physical CPUs compared to the other providers and many times more RAM.
I think you might want to go back and run a web application benchmark like you did last time; the benchmarks you used this time apparently do not benefit from having more ram, and this is, uh, rather different from most (but not all) real-world loads.
Looking more at memory and price, and Linode is a winner there, too.
It's a pity that Slicehost doesn't offer a 32bit system, as that would negate most of the difference, and they are good people with a good product. But I don't want to pay extra money for fat pointers.
Main reason I switched as well. Memory usage on slicehost was immense. Couple that with the huge bandwidth overage costs compared to Linode, and there's no comparison.
Wow, that makes a lot of sense suddenly. I hadn't thought about that, but the 64-bitness is probably why I'm tearing through my 256mb slice so quickly on tasks that shouldn't really take up that much space.
The DNS tools are considerably better IMHO. Take a look at the screenshots on their site. It's pretty cheap so if you have a significant amount of hosts with Slicehost it's worth picking up a linode for a month just to test yourself.
One thing scares me about Linode: I can not find any information with regards to availability guarantees at all. Even when they talk about their "datacenter availability", they're talking about which image slots are available in which datacenter.
EDIT: Ok, I just found some availability information in their FAQ:
"What we can boast about is our commitment to resolving these issues in the quickest fashion possible. Most customers will tell you the last time they rebooted was to take advantage of a plan upgrade. 99.9% uptime, or your lost time is refunded back to your account. "
This essentially means it's worthless: they're only backing it up with paying you back the time your server was down (they'd better do that!). Looks like they don't have much confidence in themselves in that area.
sorry, but I don't think I've ever seen any virtual server hosting provider that went further than this. most don't even mention paying you back for lost time, and instead will just give a 99.9% figure without saying what the rolling time on that % is. To be fair, neither are linode, but at least they offer your money back, most don't.
SLA's are pretty silly and useless, as a rule. :) They're really only worth caring about if they can get you out of some long term contract. They'll never be good enough to counteract the losses you'll suffer from downtime.
SLA's are not useless, they are just useless for commodity hosting.
If (say) the NYSE was negotiating for a host, they would make sure they had an SLA that gave massive damages if the host failed.
But for commodity hosting, the provider writes the contract. There is no way they are going to expose themselves to your losses other than a fixed refund for the time you lost. Not just because of the costs, but because the costs would be a PITA to calculate. If Twitter goes down and alienates its customers, how would you calculate the damage?
Even "SLAs with teeth" are pretty useless, speaking as someone who's been involved in negotiating them (on both sides). Like I said, the most useful thing you can get from an SLA is a contract out. Even the NYSE is never going to recoup anything close to their real costs from an outage.
You don't ensure uptime with SLAs, you pick good providers and go for redundancy.
Additionally such agreements (essentially "insurance" against downtime losses) are only as good as your willingness to enforce them. Likely you'll be going to court over anything significant, and paying $100/hr or more to your team of attorneys.
but at least they offer your money back, most don't.
Well, my point is that this money back is basically worthless: say, at some point, your server was down for 4 hours, and you pay $100 / month. This means you would get $0.55 refund for 4 hours of downtime. I'm sorry, but to me that sounds like "we guarantee this amount of uptime, and we back it up by absolutely nothing!".
Not that I expect my damages to be repaid, not at all, but a webhost's availability guarantees are only as good as the money they put in to back it up. If they would, for example, promise to pay back a month of service fees, it would not recover my damages, but at least shows they're a lot more serious about the guarantees they're giving.
Anyway, I do not have anything against Linode, not at all: it's just that I'm wary whenever a webhost is this unclear about availability guarantees. I probably had a few too many bad experiences in the past.
No one has any cast iron guarantees. I've been on linode for about a year now, and only had 1 or 2 outages (Power failure, and one issue with a host update).
to be fair to linode, this is what nearly all providers do, if you complain, they refund you for the time you were down. To my knowledge, nobody in this price bracket has a real SLA. (at one point I had a pretty tough one myself, but I've weakened it to the point where it's not much better than normal.)
If you are designing a high availability application it is probably prudent to assume a single cluster will provide roughly 99% uptime (if the operators are good). So multi-homing across several locations and providers makes a lot of sense even for a small player.
I had clusters of hosts running in multiple data centers (to be certain of staying up) and they broke them all at once and did not email me. It drove me nuts. That said, maybe they will be extra careful making system-wide software changes in the future.
Do you believe everything you read on the internet? I'm not saying the figures are out but before you jump ship you should really try things out for yourself. Many other posters have said Linode has a 7 day trial period; maybe try that with one of your sites before switching completely (I probably will).
edit: Sorry, someone below has said similar thing.. should of read the entire thread first!
Very low storage, poor memory/$, unestablished company, but good results in a less than scientific performance comparison by a blogger. Yeah, lets switch right away. (Linode is awesome, but the reasoning is poor)
unestablished company? uh, Linode has been around longer than I have. (And I've been around longer than slicehost.) If any of us VPS providers are established, Linode is. (johncompanies has also been around a while.)
Slicehost goes down 20%? I was aware of the 10% payment credit for paying max(monthly_bill * 6, 240) in advance, which works out to be a 9.1% discount. Is there an option I'm missing? I wouldn't mind putting a few hundred in my pocket over the next year.
Linode has been awesome since I got a VPS through RailsRumble with them 2 years ago. Till this day, I still use Linode and they have the best price for VPS as well as features that you get FOC unlike slice host.
And you know what's great about them, they have been quietly supporting the Rails Rumble community and it's time we show them our support for being best at what they do!
Yeah, what they do for the RailsRumble competition is awesome. They set aside hundreds of instances for competitors during the competition (which is a wonder on its own), and they give out several large instances as one of the prizes for the winners.
I switched from slicehost to Linode after this year's Rails Rumble. It was seamless and I've had great performance. The one time I had to contact tech support, my issue was resolved in 7 minutes.
My first observation was the standard deviation on some of these measurements, Zed Shaw would have a heart attack on some of these measurements, and I don't believe they should be trusted. The best way to measure performance is to run your app on it, as synthetic benchmarks mean next to nothing compared to your actual application benchmarks. Finally, if you are this worried about performance, you should probably be looking at running real hardware anyways, like GitHub now is.
The standard deviation is simply a measure of the spread of a distribution. There's nothing wrong or right about high standard deviations. In fact, the high standard deviation means that you should expect highly variant performance.
Look at the figures. The performance of Slicehost follows a sawtooth like pattern. The quantity standard deviation is useful because it quantifies what to expect. Plus or minus one standard deviation means that ~ 2/3 of the time you will fall in that range.
If you think about the problem a little bit, you might be more worried about the standard deviation of the standard deviation. This, in fact, would be a useful quantity, but hard to measure.
EDIT below this line -------
Several comments below have commented that SD is somehow less useful if it's "large" (or large relative to the mean, or whatever). The reason people think large SDs are indicative of a poor experiment is that in school lab classes one calculates the SD and call it the "error".
The standard deviation is a measure of spread, if it's large then the spread is large. Knowing the spread has value. In this case, under the parent's experimental conditions EC2's performance is more constant than that of slicehost's.
A fair critique of the blog posting is that the error on the standard deviation may be large, depending on the experimental conditions. It is _not_ a fair critique to say that the SD is too high to make a prediction, you just have larger performance spread. Note that the performance spread described is not necessarily "error". The spread is inherit to either the server (as implied by the article) or the method (in which case it is an error).
I understand the performance variance characteristics, what concerns me about these tests is that the graphs of them are not continuous, they seem to be immediate dips instead of gradual curves. This indicates to me that the sample size wasn't large enough, or at least the graph needs a higher resolution if data exists to support it. Also, at the bottom, the article gives the numerical data, with the mean and standard deviation. However, the mean for the hosts with high standard deviations are essentially useless, because the standard deviation is so high. If we are going to compare average performance between cloud hosts, lets at least have useful averages to base our opinion on.
I think that low standard deviation / variance is desirable. It provides a level of performance that can be predictable and reliable. Then just multiply according to your processing needs.
Somebody please do this for the low-end dedicated providers. You can get Atom systems for $30-$50/mo now, which should solve the performance variance (especially on disk IO) that affect cloud/VPSes.
Provide network latency/tput data also, please. :-)
The price for dedicated servers in Germany is just crazy ... a friend of mine is using this one too and I also have an account. If price is a concern (for testing and personal purposes), find a friend or two to share a baseline dedicated server is usually much better than using a cheap VPS.
I signed up with Slicehost a few months ago. It was my first experience with VPS hosting and Unix.
As a beginner, Slicehost was really great. The control panel is really straight forward and easy to use, their support is quick and helpful, and their how-to articles are really great for beginners.
So they both have articles for the absolute beginners, but it still seems like Linode slightly favors the more intermediate to advanced users in their how-to's. Purely subjective, but Slicehost's seem a bit easier to read / follow.
That being said, this article was incredibly interesting. Now that I have a bit more experience with VPS's and Unix, I am very tempted to head over to Linode.
I am still developing my web app on Slicehost but I have not seen any performance problems since I have very little traffic. I am now considering setting up the production server over at Linode when I'm ready to go live.
Why does Prgmr offer so much more memory/$ compared to the other providers? Is there something about the usual workloads that makes RAM less important or is it based on hardware capabilities?
As far as I can tell, the amount of ram is more important than anything else when it comes to real-world performance, but it doesn't really feature in most synthetic benchmarks, which I believe is why I did so well in Uggedal's first benchmark, and so poorly in the second.
I have dealt with Nick a few days ago and I found him one of the most competent, friendliest and patient tech support people I have ever dealt with. 4 hours of his time dealing with my newbie shenanigans when I am only paying you $20/mo; that alone has won me over and I have been a walking Prgmr PR agent ever since that day.
To everyone who doesn't know: #prgmr channel on Freenode
I was once contacted about an old ticket and asked by Will if it was solved, since it was solved on their IRC channel (and was not marked as solved maybe). It shows that they make sure they respond to all the tickets. Really sweet of them to do it.
I agree with mahmud. Same here friend, I've been that "walking PR agent" for prgmr ever since I started out with them.
Besides Prgmr is for those who like the real raw stuff - bare bones VPS with shell access.
The SLA is always something you don't want to read with any hosts :P Coz it includes all the bells and whistles they say they would provide you if they go down,and that would sound like they would go down :)
There were 3 reasons why I use Prgmr:
1.) They support Paypal (yes, I cannot use a credit card since I'm not yet eligible to have one in my country)
2.) The pricing is great
3.) They have this small 128mb pkg on which I can develop and test my app. So when launching I can upgrade my plan. Thus saving me a few dollars which might sum up to one more month's payment when I launch :)
+1. though not fully committed mytime to move my services and stuff to prgmr servers yet, still it is/was much more fun and trust building when communicating with a real person while trying to solve issues.
99.5% seems bad until you realise the rolling period is a month. that's pretty good actually, since most will roll over a year, or not mention roll over at all, leaving them fine to say 99.9% covers 3.65 days in a 10 year period if they want. can you imagine your site being down for 1 day and they say it's within contract :)
my cpu is probably worse 'cause I only give you 1 VCPU. I don't know for sure this is the right decision, but the idea is that the fewer VCPUs each instance has, the less often you have context switches, and the better your system performs when there is a lot of contention for CPUs.
I'm not switching because they meet my needs, my customers are happy, and switching would take me lots of time that I can more profitably use promoting my software. I don't get paid for doing work, I get paid for solving problems. Slicehost is not a problem for me.
Well put :) I asked the question because server admin is probably the area where we're weakest and I don't want to waste time trying to solve problems that could easily be solved by switching to another hosting provider.
Benchmarks is benchmarks, you should take them with a grain of salt. Sign up for a Linode account and test how it fares on HTTP requests (siege/ab) or test suite for your own application.
Well, I do not switch because my slice keeps being online and available for almost two years now. "Never change a running system" ;).
I do, however, also have a Linode VPS and will probably go with additional Linode VPS' in the future if the service continues to be stable.
In my humble opinion, Slicehost offered a great deal two years ago but they should adjust their plans in the future or customers will move to competitors.
I'll probably not switch (immediately) because of inertia. I've got too many systems to admin on my other network and if anything i'll just migrate over to some hardware that I have. Afterall, I just host a blog now on Slicehost :)
I would like to see a comparison of the APIs provided by these companies.
My understanding is that Amazon is competing on developer and sysadmin productivity with the depth of their API and focus on manageability-- not on performance. Is that correct?
I think the big advantage of Amazon is that you can scale up and down quickly so that you aren't paying for services you don't need. They also have the advantage of better integration with their other services since you don't pay for transfers within the same region and the transfers are fast since it's all on their network.
I wonder if the Slicehost numbers were generated on an older slice? He did note that the writeup was six months in the making. If he was on older Slicehost gear and the latest Linode gear, that would make some difference, at least it seems like it would.
At one point I had the Slicehost crew set my account to provision slices in their newest datacenter in Texas. Things seemed snappier after that, though there could be several reasons why and it's hard to know if it was just a matter of newer hardware running the Xen dom0.
i have two linodes, one for high bandwidth (v2d7c.sheepserver.net) and one for low (electricsheep.org and scottdraves.com etc). the admin tools are awesome, there is hardly any (if any) downtime, performance is great, and prices are low. two thumbs up.
If I remember correctly, Xen will give your instance more CPU if other VMs on the same physical host do not need it at the moment. If that is indeed the case, then your CPU performance comparison isn't very useful, as it only shows that you were lucky to get an otherwise idle machine with some providers, while others were loaded.
It would have probably been better if these tests were run on different host machines. An easy way to have accomplished this would have been using the migration feature to switch data centers, etc.
I skimmed through the comments again. I find it funny how people jump to conclusions, focusing on providers, and thinking about switching.
In my humble opinion, at least some of these benchmarks (the CPU ones) do not represent anything, as they depend on the load of neighbor instances on the same physical host. Once you see that, how can you possibly make business decisions based on these benchmarks?
Before anybody goes and switches hosts, be sure to consider the cost of actually moving servers. It does take quite a bit of time to move servers, and while it may be worth it in the long term, it may not benefit very many people in the immediate future (I'm not saying definitely don't do it).
If you are running a site that is used by others and you don't have a system to automatically rebuild your server on another host, I'd say you have bigger problems than the relative performance of your provider's architecture.
For people like me who don't need a lot of performance is more important other issues like support (quality and response time), backups, documentation, provisioning, uptime etc.
I'd like to see a more complete comparison. I personally use Rimuhosting and Rackspace, I'm happy with both.
I have used slicehost for a long time it served me well. I don't use them anymore. One of my favorite option was the ability to increase the size of slice whenever I needed it and get charged only for the time it was used. Is it possible to do the same with Linode? As often, quickly and easily as I want?
If so, I can see myself getting Linode for my next project. Heard a lot of good things about them in the past.
In defense of slicehost, they have a no-bs interface with good reading resources to setup machines and very fast and helpful customer service. I was once more than 10 days late in payment because I had problem transferring my money to the account I exclusively use only for online payment and they were very reasonable with me.
Linode will charge you at the beginning of your billing cycle and if you ever resize, which you can do automatically through the control panel, you will receive a credit for how much you didn't use.
I wish your study accounted for the elasticity. I end up using 20-30 machines (sometimes 100) for couple of hours every day. Normally, I would use 3-4 large instances.
I wonder how EC2 compares with others in scenarios similar to above.
I'd also be really interest to see network performance tested too. Instance->Instance, Cloud colo->cloud colo, cloud->user and cloud->other clouds/web services.
I'm using a small EC2 instance to run a service that used to run in a very small real computer. This instance is unbelievable slow. I'll switch ASAP. Benchmarks are benchmarks but EC2 is slow and overpriced.
This is an apples to oranges comparison. Amazon's "small" instance is far slower than any other instance type they offer. A medium instance, for example, costs twice as much but is much more than twice as fast. He also compared it to all x64 instances; why not use a 64-bit Amazon instance?
If he put an Amazon medium or large instance up against those others, it would have fared much, much better.
If you read the article I ran the benchmarks on both i686 and x86_64 on Linode (which gives you a choice). And paying more than $120/month for one instance is not within my budget for http://wasitup.com which I did the comparison for.
I'm sorry but this seems to be one of the many flaws in this benchmarks. Kudos for your efforts but the graphs are almost meaningless, all things considered.
For example the EC2 instance storage is known to be dog slow. I assume that's the main reason for it looking so bad in your graphs - the picture might change with EBS.
As others have pointed out, synthetic benchmarks are a tricky beast generally. For fairness you should have optimized each host to their max potential - because that's what a regular user would do. But still, the variance in cloud-hosting makes it difficult to obtain representative figures. Slicehost might just have looked awesome in a different week...
Exactly, I would claim the author to be a fool if this wasn't included.... we are all entrepreneurs here, right?
In the same way I never batted an eyelid with David Welton's Slicehost vs. Linode[1] article. It was informativve and unbiased and I was happy to provide some incentive back for that.
Now, 4 hours in I have 3 Linode referrals. If these stay customers for 3 months I'd get one month of free hosting for http://wasitup.com (which I provide free for all).
When you knock 4 other companies and promote one, then make money off people buying the service based on your analysis, you create an impression of being non-objective.
He is not giving you his opinion, he is giving you results of his benchmark. You can even download logs to check and run the patched benchmark util yourself to compare.
I (and most others hopefully) already knew about prgmr and EC2 has horrible performance, even before I saw this benchmark, the only one new to me is Rackspace performance numbers (BTW Rackspace owns slicehost).
The overall presentation convinces me this is a good-faith effort, but such a financial interest in the results would ideally be disclosed prominently up front.
Even with no conscious/devious intent, this sort of interest can bias the results. For example, what if the relative results could vary arbitrarily from week to week (especially possible with the neighboring load on VPS nodes)?
If you get a plausible result that shows the companies pay commissions do best, you are incented to publish right away. If you get a result that shows the non-commission-paying companies on top, you will dig deeper: maybe there are ways the commission-payers are better, in a longer analysis.
(It's similar to the publication bias in other research, which also requires no scheming intent, just natural behavior in response to incentives which are each alone reasonable.)
Generally I agree with you, on the importance of being skeptical about analysis/benchmark comparison where financial incentives are at sake. IE, some research publications are sponsored by Microsoft who in return publishes glowing review of Internet Explorer 8.
However, in this case, the author is not sponsored by or works for any VPS service. He provided affiliated links at the end of the article, where one of the linked service did not prove to be superior than the other according to his analysis. If he sneaked in an affiliated link within the article, the objection and skepticism could be somewhat justified (still I think its somewhat fair game).
Yep, the reason Slicehost and Linode is mentioned at the bottom is since they are the only providers which had a referral program (as far as I could gather).
