"If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it."
That's what I would call a mortal wound to Verizon's arguments.
Clever technique to draw attention away from what is most likely a contractual dispute.
Verizon's argument is clear:
> Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows
Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse
> in this case, since Level3 is pushing more data, they would be the ones paying
No. That's not how the internet works. Nobody PUSHES data. People PULL data.
Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data.
They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase.
If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it.
This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.
1) Level3 has offered to split the cost with the Big5 ISP's of upgrading their infrastructure.[1]
* This would increase throughput and capacity for both Level3 and the consumer networks.
2) Netflix has offered free Open Connect Appliances to all major ISP's.[2]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all their customers. In-network traffic is almost free for the ISP's.
3) Netflix has offered to change their entire distribution model into a P2P model.[3]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all customers. In-network traffic is almost free for ISP's.
.
All of these options have been flat-out refused. The Big5 ISP's are purely after the money. There is no other compromise for them.
I work at an ISP, can confirm. All decisions about capacity are about how much money we'll make or lose. We're intentionally keeping bandwidth constrained because we know it will severely impact our cable TV business.
And that is the main problem here: ISP's are competing with themselves. This is why everyone hails Google Fiber so much: it's a dedicated internet service provider, and does not compete with another branch of the same company. As a result, they (or any dedicated ISP for that matter) will try to get their users the best internet experience and speed out there. In theory anyway.
Thought of a great analogy. Imagine it's the nineteenth century, and by whatever coincidence the dominant electric power delivery company actually also still owns a thriving kerosene delivery business, i.e. for lamp oil. They don't care if you run sewing machines or home appliances at the going rate for electricity - but every night after dark, they determine whether you're streaming electricity to power your light, and if so they intermittantly dim the power (to well below the level you're paying for), expressly to give you an awful experience and retard the adoption of electricity. Just so they can make more money selling lamp oil.
Net neutrality simply means they are not allowed to go into homes and figure out what you're using your electricity for, and try to charge you more based on that usage.
That's true, and an exceptionally strong argument for forcing ISPs to be utility providers that are not allowed to have content businesses. It's back to Econ 101.
The problem with this is that to some extent, the content business effectively sponsors the internet business. The same is true for the phone business.
It gives them pops, "free" capacity (meaning they have to buy dark fiber, and can resell internally cheaply, or just wouldn't have it at all if they didn't have phone business), shared infrastructure.
So if you do this, effectively it will mean the same any other solution means : more expensive internet.
Given that that will be the result, might I suggest an alternative solution ? Get a basic business internet line from Verizon. See the difference.
The bulk of your "business internet line" services are really only expensive due to the SLA that is attached. Drop your SLA, and the price will drop sometimes up to 80% (even though it is still technically "business internet connection" and is on the same business line subnet, etc).
Nobody says ISPs should ignore short term profit/loss considerations. However, more and more people are saying by ignoring everything else, they are making their customers and themselves a disservice. This is a dangerous game, and a business which thinks they can make money while pissing off their customers is putting itself in danger. Sometimes it can be sustained, but in many cases in the long run the customers would either leave or set the regulators on them.
Netflix could solve the bandwidth problem by just mailing the whole library to customers. "Never underestimate the bandwidth of a station wagon full of magnetic tapes."
Netflix offer this as an option to any ISP that wants it, free.
They will put a server in a data center anywhere an ISP wants, and they'll manage the whole syncing thing so the ISP doesn't even know it's there. After that, all content comes from that cache.
To clarify Grecy's statement, netflix offers openconnect[1] for to ISPs as a caching mechanism, and the box contains about 108TB of disk capacity[2]. The boxes are sent to ISPs instead of customers though.
While the "oil tanker full of hard drives" approach has incredible bandwidth the latency is a bit of a problem. Especially when the Netflix library is updated constantly.
My personal opinion is that you should cache/co-locate your data close the the edge of the edge of your network whenever it presents a benefit for congestion and price.
Its the senders responsibility to deliver data to the local ISP of the recipient. It doesn't matter if the person asked for the data, you still have to pay to get it there.
The issue is that Verizon is both a national transit provider and a local ISP. If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain. You are asking Verizon to do it for you, as a transit provider.
However, if you are delivering it to the last mile, you have "paid for it" it already.
The issue is where are these congestion issues occurring. At the last mile or at the edges of Verizon's transit network.
Verizon absolutely has the right to bill for providing transit.
"Verizon absolutely has the right to bill for providing transit."
Even if we agree that Verizon has the right to bill Level 3 for transit within _Verizon's own network_, it seems odd that they are not taking advantage of Netflix's CDN boxes, which would drastically reduce the amount of transit required.
If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain.
So, let's say I build a new skyscraper with 10,000 apartments, and I set myself up as the exclusive ISP to the tenants of that skyscraper. I wire up the building nicely, and have some big switches and routers in the basement. Does every content provider (e.g. Netflix) or company working on their behalf (e.g. Level 3) now have an obligation to build new circuits, and deliver the traffic all the way to my basement?
If this were telecom/PSTN, yes. I know this is not the PSTN, but bear with me for a second.
For years, this is how Verizon/AT&T/SBC/BellSouth/etc had structured telephone networks. And this is probably the same structure they want to extend to shared data networks aka the Internet.
If you were a competitive telco carrier who wanted to deliver a massive amount of phone calls to Verizon customers in a particular region, you couldn't just dump it off at the tandem (which you can think of like a telecom peering point), you had to extend your network to the end offices where the Verizon customers were aggregated.
The thought there being there's finite capacity between the end office and the tandem. And if you're going to use most of it, you should either pay for it or build your own facilities to alleviate congestion.
You're dealing with a traditional telecom company (Verizon) in its telecom mindset (build to me/mid-span meet, and keep a very tight watch on ratios/meet points via accounting, billing, and state regulation), vs. a traditional Internet company (Level 3) in its traditional peering mindset (build to the exchange, then build to me, and keep a loose observation on ratios/meet points with no regulation).
So, let's say I build a new skyscraper with 10,000 apartments, and I set myself up as the exclusive POTS/PSTN provider to the tenants of that skyscraper. I wire up the building nicely, and have some big switches and routers in the basement. Does every other phone company now have an obligation to build new circuits, and extend their network all the way to my basement?
As a registered telecom provider with the state, you need to file at least one interconnection agreement (the telecom equivalent of a peering agreement) with the regulatory body showing that you are connected with the incumbent (Verizon) or someone who's going to transit calls for you to the incumbent.
The state won't let you take on calls as a local access provider if they're going to end up going nowhere.
You also have a regulated obligation to provide working 911 service so you need to interconnect with the incumbent's 911 tandem switches. So, off the bat, you have multiple connections with or to Verizon.
Now, if there are some really popular people in your building who attract a certain amount of calls, YES, you can demand that the other telecom carriers extend facilities to your switch (which is located in your basement).
Those carriers will decide if they want to build facilities to you, or work with another carrier (remember the one interconnection agreement you had to sign?), to pay them to deliver calls to you on the agreement that they have enough facilities with you.
You are compensated, per minute for any call that terminates on your skyscraper network through a process called reciprocal compensation, whose rates are laid out in the interconnection agreement you signed.
Is old-timey Verizon's attitude starting to make sense?
So Netflix should pay Verizon $1/month/customer. And pass that cost on to the Verizon subscribers...
How quickly would this change Verizon's policy?
As a customer of an ISP, I pay for access to servers on the internet. My ISP charges me for a level of bandwidth. If they are the bottleneck slowing down my connection to the services I want to access, they should fix it, and give me what I'm paying them for.
My contract says I'm paying for access to the Internet. Access to the Internet involves transmitting data outwards, and receiving data inwards. So yes, I am paying them to transmit L3's traffic.
No it's not. You pay to send to them, and to receive data people send to you. You don't pay for your ISP to transit data from whoever is sending you data.
For example, you want to download something off a Server in Germany. Your ISP is responsible for getting your initial message to the server. But it's not responsible for taking that information in Germany and carrying to back to the US for you. The Server's ISP has to carry it to the USA and then dump it off at your local ISP.
Now your ISP and the Server ISP may have a peering point in Germany, but they might not. But either way, your ISP isn't under any obligation to pay for transit of data being sent to you. That's just not how it works. And that definitely isn't what is in your agreement with your ISP.
The question then becomes does the contract Verizon have for peering with other transit providers allow them to charge the other transit customers for that? And if they do what's to stop every transit provider from charging every other transit provider's customers for the same thing? And then the all important question, SHOULD they be stopped from doing it? The current structure of the internet was built around this not happening that way, but is that the way it should be? I strongly believe that letting it happen would cause many other political and routing problems for the internet as a whole to the point where it'll end up being regulated against doing it that way anyway. It could also end up then as a zero sum game between all the transit providers ending "free" peering agreements.
It's a fair distinction. Here, I believe, the movie is going AWS Seattle to L3 to the front door of Verizon's consumer network in LA, and Netflix is paying for it all the way to the front door.
Normally I pay for transit from my house to my ISPs peering points. And other service providers pay their ISPs for transit from their "houses" to the same peering points.
In this case Netflix is being forced to pay for transit from their "house" all the way to mine. Not how the internet normally works.
AWS may be the origin for content distributed via peering, appliances, and CDN partners but your videos aren't streaming from AWS: https://www.netflix.com/openconnect
they are the poster child for using AWS for what it is good for, general purpose compute (storing your play list, the list of all movies, controlling the CDN etc)
You still haven't refuted the point. This isn't about the relationship between Verizon and its customers, at all.
The agreement between Level3 and Verizon probably has some balanced traffic requirement, with penalties for asymmetry. If that is the case, then the nature of the imbalance would force Level3 to make payments to Verizon.
Verizon gave a very clear suggestion for Netflix: work with other providers as well. If Netflix worked with the overwhelming majority of third party providers, then you could argue that Verizon is not playing fair. But I don't see any commentary that Netflix is actually working with most of the third party providers.
ISPs charge me primarily to download. Literally I can't upload as much as I download. My uploads are contractually limited to 20% of my downloads.
So does that mean there's a downloader-pays rule?
If so that means that Verizon should be paying Level3 because it consumes 3x the bandwidth that it provides.
Verizon's needs Level3 3x as badly as Level3 needs Verizon on a bit-by-bit basis. So why isn't Verizon paying Level3 for the imbalance?
There are actually two sides to the ISP game. There's the consumer ISP side and there's the commercial ISP side.
Consumer ISPs are in business to get their customers hooked up to as much of the internet as possible to enable them to consume things. And they charge their customers (normal folk like us) for this service.
Commercial ISPs are in business to get their customers hooked up to as much of the internet as possible to provide services. And they charge their customers (big companies) for it.
When commerial ISPs meet consumer ISPs typically they say "let's trade traffic for free so that way we can both keep our money and not worry about accounting"
What's happened is that a consumer ISP is trying to charge more than just it's last-mile normal customers for data that they use and which it is ostensibly contractually obliged to provide to them without further fees.
What Verizon is saying is "look they need to pay us to upgrade this link" which on the surface doesn't sound too bad. It kinda makes sense. If I used more bandwidth they would charge me more. But the thing is that I am already paying them to make sure that the connection to the internet is uncongested. They are deliberately allowing it to get congested to try and bill both sides of the transit.
They may have a peer balancing agreement, which is not being held up, but they're also still being payed by their customers to provide access to content outside of their own network and Verizon is enforcing a limitation from another agreement (the peering agreement) that is affecting their ability to provide the content their customers want to have access to.
You can't have it both ways. Or... I guess you can, as Verizon has shown.
Commercial ISPs vs. Consumer ISPs is a useful way to frame the issue. From an economic perspective, the big difference is that the Consumer ISPs generally have a monopoly over the Consumer endpoints, while the Commercial ISPs do not have a monopoly over Commercial endpoints and must negotiate a competitive market.
This obviously gives the Consumer ISPs pricing power in all of their business dealings (including any peering contracts that they force the Commercial ISPs to sign). The fact that Verizon owns RedBox (a Netflix competitor) & sells TV packages is just further motivation for hampering the interconnection.
Yes I'm well aware. A good friend of mine works on the commercial side. He is a programmer that is basically in charge of all their route-finding algorithms internal to the commercial side of the network. It is completely separate from the consumer side.
On the commercial side they provision a link from point A to point B (which is what his software figures out) and it's dedicated bandwidth. They are never over-subscribed.
I'm almost positive that this is not how they do things on the consumer side. Otherwise the Netflix debacle wouldn't be happening. But basically the consumer side is always over-subscribed and by a reasonably large factor. They're counting on consumer traffic to be bursty which it normally is. Netflix isn't bursty the way they're used to.
I sympathize with the idea of expecting one thing when building a model and seeing it play out differently. It sucks and it can cost a lot of money to make good on promises that you've made. But remember that Verizon is the one that made these promises. Their customers didn't put a gun to their head and say "promise me these crazy high download speeds or else!", Verizon did it willingly to try and steal customers from their competition.
Because they did it to themselves I have little sympathy.
Netflix debacle is Netflix choosing not to pay for CDN and instead asking ISPs to take thier CDN for free or we overload your interconnect ports and blame you for it on our site.
> But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download -- thus filling their upstream networks and nearly doubling our total traffic -- there is an uncomfortable silence. That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs.
> in other words, moving to peer-to-peer content delivery
Verizon is charging me for access to the internet -- then charging the internet for access to me. That is double-billing, and it's egregious.
Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity. This is the same company that over-subscribes their network on purpose, as most ISP's do. They will come into a neighborhood with a 1Gbps link, then sell 50Mbps to 100 homes, in hopes that not everyone uses the connection at the same time, nor for long duration.
This is a weak argument. Internet connectivity simply is not measured as a 1-dimensional value. It is effectively infinite dimensionsal.
The marketing papers over this in misleading ways, and that is bad, but if you wanted to pay for your own dedicated bundle of wires, like a businesses office building does, you could get it.
>The marketing papers over this in misleading ways, and that is bad
It's not a weak argument at all and the misleading marketing is exactly the reason it is not. In fact, Verizon has oversold bandwidth (or at least offered it for less than they are willing to accept).
But, what you seem to be effectively saying is that customers are at fault for buying an advertised product which touted the benefits they desired at a price that was acceptable.
> Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity.
If your last mile network was built to allow you to saturate your 50Mbps connection 24/7 your bill would be much larger than it is now. So much that you probably wouldn't pay for internet service, so oversubscribing is a totally normal and rational process when building out a network.
If I accept your premise as true, then please explain:
1. Kansas City, KS
2. Chattanooga, TN
3. Provo, UT
4. Austin, TX
These places all have fiber to the house, and supposedly gigabit speeds. Two of them were built out with zero help from Google, so Google can't be making huge losses on the internet to prop up other parts of their business.
I'm not sure how you are misunderstanding or purposefully ignoring what I'm writing in every post.
Your network no matter the bandwidth to your house is oversubscribed. If everyone was using their network connection at max capacity 24/7 there would be network congestion issues.
Thus the arguments that "I've been sold 50Mbps/down I should be able to use it at max capacity 24/7" without issue are incorrect.
> Your network no matter the bandwidth to your house is oversubscribed. If everyone was using their network connection at max capacity 24/7 there would be network congestion issues.
No, not necessarily. That's how Verizon Residential chose to build out their network but that's not the only way to do so. The commercial side of Verizon manages to provide their customers with 100% of SLA throughput 24/7 no problems. Technically this is feasible.
I would agree that I might not be able to get 100% utilization at the price I am currently paying. That is a point I will concede. BUT, SOMEHOW Google Fiber in three cities and Chattanooga is also supplying gigabit fiber. And they're not charging $10k/mo per person to do so. Could everyone use the full gigabit connection 24/7? I seriously doubt it. But if they're oversubscribed by 10x on gigabit and other places are oversubscribed by 10x on 100 Mbps then they are successfully delivering 10x the internet for the same price, or perhaps 50%-100% more. It's not linear that's for sure.
For your argument to hold water if Google Fiber was 10x faster than Verizon, then Google Fiber would also have to be 10x as oversubscribed as Verizon. I very, very seriously doubt that this is the case.
> Thus the arguments that "I've been sold 50Mbps/down I should be able to use it at max capacity 24/7" without issue are incorrect.
This is a straw man argument. You are not engaging in an honest dialog. You seem to be purposefully obtuse. People watching Netflix in NO WAY constitutes 100%, 24/7 utilization!
How about "I've been sold 50 Mbps service and I want to use 3 Mbps of it for an average of 3 hours per day, which is a measly 6% of the advertised speed and is only 12.5% of the 24/7"
There are two ways to do the math here. One is based on peak utilization, where we assume that everyone watches Netflix at the same time, 100% identically, say from 6pm-9pm. In that case Verizon can be over-subscribed by a factor of roughly 16x (50Mbps/3Mbps) and still keep their customers happy.
The other is to assume that everyone watches Netflix perfectly randomly. In that case let's assume 3 hours per day. Here Verizon can be over-subscribed first by the 16x (since people don't use all 50 Mbps) and further by another 8x (since people are using the service for 3/24 hours per day). That gives you an over-subscribed factor of 128x.
The truth of course would lie somewhere in between 16x and 128x. But what we're seeing is that Verizon is over-subscribed by a factor of more than 16x otherwise people wouldn't have any problems watching Netflix. Put as percentages instead of ratios, customers are asking Verizon to provide between 0.75% and 6% of what they promised to deliver, and Verizon is refusing to do even that.
Further we have evidence via a blog post from Verizon that their entire network is NOT oversubscribed by a factor of 16x as it's only at 40-60% utilization. What is happening is that Verizon is throttling their users at the peering point. It's a great gig if you don't get caught. http://publicpolicy.verizon.com/blog/entry/why-is-netflix-bu...
If we assume that 100% of all traffic on Verizon's network is from Netflix that means Verizon could double the peering bandwidth to L3 before their network got congested. Is that a valid assumption? I don't know, but I sincerely doubt it's 100%. During peak hours it might make up 50%. In that case doubling the peering bandwidth would increase utilization from roughly 50% to roughly 75% which is perfectly acceptable.
It must take some pretty fantastic mental gymnastics to convince yourself that a provider who is not able to make good on 6% end-user utilization is totally in the right and their upset customers are irrational and wrong.
Which is not how it works in the rest of the world. Only U.S. consumers are so used to getting screwed that they would defend it as normal. Verizon loves double-billing so much for cell service that they want to do the same with the internet.
There are other countries that bill for incoming calls. I know of at least Singapore that does it.
Also, your phone provider does get charged for each incoming call. They just eat/redistribute that cost, since for normal phone user it's likely to be less than a penny per month. But try running a conference service and you're definitely going to get billed for incoming calls (apart from some weird force-subsidised areas where you can get paid to receive calls)
The difference is that in Singapore, you can get charged for incoming calls depending on your contract, but you _don't_ get charged for incoming SMS.
From a Singaporean perspective, that's the most bizarre part of mobile service in the US. I need to get an unlimited texting plan so that I don't get billed for people sending me texts? Whoa!
In India, when we were younger, we'd sometimes all call someone on their birthday. Before you know it, there'd be eight people at a time (with people dropping in and out to wish) on a conference call held at the birthday person's number. That person would pay nothing for this.
U.S. consumers think it's extremely unfair that a poor guy with a landline should subsidize the rich guy with the cell phone or the satellite phone. It might cost me $1 a minute to call my neighbor? No thanks!
Well, that is a damned strange US custom that isn't the case in many other places. It is certainly not the case here in Europe, for example. The receiver pays.
No one pays for incoming calls in India. This is one of the reasons why even very poor have cell phones. Unfortunately american cell phone users aren't that lucky. Both the caller and callee have to pay. Verizon being cell phone company probably want to charge both sides. Most likely they will win.
This is a very bad analogy. When you sign up for mobile phone service, you're also agreeing to usage fees or paying extra to get "unlimited", which case, you pay nothing for those calls.
When I sign up with an internet provider, I am agreeing to pay a fee for open access to the internet at large. I'm not signing up to pay for each piece of content I access or each site I connect to.
I've never agreed, to the best of my knowledge to pay for access only to content served at a rate limited by the reverse utilization of the peering connection that content provider's content can be accessed across.
To use your own analogy -- it would be as-if your phone provider charged you to call me (they do), my phone provider charged me to receive the call (they do), then my phone provider charged you for making the call (they don't).
Actually, they do (on that last point). Which is why Google Voice won't route calls to certain numbers. It's called a termination fee, which works like this: I pay my local phone company, and also pay a long distance carrier. When I call you, your local phone company charges the long distance carrier a fee, which that carrier then passes on to me. (Well, it used to work this way, but almost everything is flat rate now).
If you are using a cell phone or a satellite phone, you pay for your connection to that network, whether or not you started the call.
You certainly don't make it so that grandma using her POTS has to pay for the guy with the more expensive connection. That is viewed as extremely unfair.
OTOH, using the cell network generally has free calling to anywhere in the lower 48. It's not the old physical wires that are the expensive part, it's maintaining and upgrading all the cell towers, and so people pay for that usage. Once you've paid to connect your call to the wired network, you can do whatever within very wide geographic borders.
It's actually a historical issue. In the US cellphones orginaly used standard POTS numbers while in other countries cellphones often had an easily distinguishable format. This means that in the US granny wouldn't know she was calling a cellphone so charging her extra would be an unpleasant suprise, while in other countries she knew and was thus implicitly accepting to pay the charge.
I know the answer to this one! We need these guys to setup at the other end of a Level3 link, and then Verizon customers can push them one byte for each byte they get from Netflix:
Seriously though: what if Netflix worked with Level3 to set up an efficient Level3-owned /dev/null server in the carrier hotel AND have every Netflix client that connects over Verizon upload random bits to that IP address whenever buffering is complete or the movie is paused? Then the rates would balance out, and Verizon would no longer have a contractual excuse to NOT hook up those cables to the Level3 endpoint. The only way for Verizon to mitigate this would be to deny their end users the upstream bandwidth they've paid for but are not using (cue user outcry), or to smear Netflix even more so than they're already doing (cue more chances for the PR to backfire). Verizon would hate Level3 for this, but they already do, and they couldn't do anything about it without denying their end users Netflix access altogether!
Of course, since the NSA would need to listen in on all that new random-bit traffic (since Netflix is obviously the new Skype in the terror playbook), taxes would need to rise significantly in order to pay for more NSA bulk-data-collection servers. But that's just the cost of living in America...
Verizon gave a very clear suggestion for Netflix: work with other providers as well.
Why should they have to do that? Level 3 has open ports now (so they claim). It doesn't matter which provider the traffic is coming from, either Verizon wants it or they don't (and apparently they don't, even though their core is no where close to saturated).
The balanced traffic requirement is bogus anyway. Its not as if Verizon offers their own video streaming service that I can subscribe to as a Comcast or AT&T customer. So they're never going to see balanced traffic in the age of video streaming.
It is entirely about the relationship between Verizon and their customers. Verizon's customers want these interlinks, they just don't know it. This whole thing started when Netflix posted a message to their customers explaining it. Verizon is trying to defend themselves against the complaints of their customers.
The concept of asymmetry as you're using it doesn't make sense in this situation.
- If you push traffic to my network it is because someone in my network requested it.
- Incoming and outgoing traffic are technically no different.
- It makes more sense to pay to receive traffic than get paid to receive traffic. That's actually what VZ customers pay for and that's why not many people complain about ADSL speeds.
If I were Level3 I would actively partner with VZ alternatives in the US, announce it to VZ customers so they can move away and then proceed to remove any peering with VZ. Enough bullshit already.
I don't think in the history of the internet there has ever been a "balanced" traffic arrangement. The majority of people are always going to be requesting more than they are uploading.
As a residential subscriber this is contractually the case. ISPs do not allow the operation of servers in residential contracts and they are almost always provisioned with asymmetric speed. 50 down, 5 up, etc.
Instead of double billing (i.e. subscriber + content providers) couldn't Verizon just increase their subscriber rates to cover their perceived increased costs?
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts? Claiming that Verizon charges Netflix at extra e.g. $1/subscriber/month to peer?
Would it be legal for Verizon to literally just cut off Netflix? Just return an access denied to their subscribers.
(3) I actually don't know off the top of my head.
(1) would certainly be doable, but it's something Verizon wants to avoid because in most markets they're in a duopoly (often with Xfinity, sometimes with a local provider). Much like two gas stations across the street from each other often having identical prices, Verizon doesn't want to be perceived as costing more for the same service as the alternative.
For (2), Netflix could almost certainly do that, but I assume they'd prefer to not pass any cost onto consumers (because, really, they're making a stand against this business practice by Verizon being fair play at all). Among other reasons, if it's seen that Netflix is willing to pay a "ransom," other carriers may try to increase their contracted costs.
To your two pricing questions, the answer is "sure, but":
1. Verizon, as a monopolist, is already pricing its service at the level that extracts the most revenue while remaining within the envelope of whatever loose regulation they have.
2. Netflix, as a company in a competitive market, can't easily increase prices without losing customers. If Netflix charges Verizon customers $9 and Hulu is only $8, it makes Netflix look less attractive, and they'd lose customers.
Agreed! But they're closer to interchangeable than say 50/5 Mbps cable service and 56.6kbps dial up. Which is some places is your only alternative to Verizon.
Further Netflix isn't the only place to get movies. I can get movies from Amazon (streaming or DVD) Walmart, and other physical stores. I don't have that same kind of choice in ISPs though.
IPSs have more of a monopoly on their subscribers than video intermediaries do. The studios can release movies through a half-dozen channels. I would (figuratively) KILL for the chance to pick from a half-dozen ISPs that have offerings which are reasonable for the modern age.
Respectively, yes, yes in the absence of network neutrality, and yes in the absence of network neutrality.
We think of ISPs as selling us data, but the big ones have realized they can make more money as data market makers. This whole kerfuffle is just them trying to see if they can get away with it. Because they're effectively monopolies on the last mile, it's important that they fail here.
couldn't Verizon just increase their subscriber rates to cover their perceived increased costs?
Verizon already increases prices x% per year, so maybe. But for every price increase there are some number of customers who will cancel or downgrade to a lower plan. (Alternately, Verizon could decrease their profits.)
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts?
Sure, but some percentage of Netflix customers would cancel.
Would it be legal for Verizon to literally just cut off Netflix?
Considering that the FCC is debating Net Neutrality right now, that would be a very bad idea.
> That's not how the internet works. Nobody PUSHES data. People PULL data.
There's more to "the Internet" than just "the WWW". While HTTP could be considered a "pull data" protocol, there's plenty of application protocols which involve "pushing" data over TCP/IP transport.
Yes I'm aware. But the vast majority of traffic happens as a result of a request of some kind.
In other words Netflix doesn't start streaming data to me for the hell of it. Pandora doesn't stream songs to my computer unless I hit play. Songs don't get downloaded from iTunes unless I request them and pay for them.
This is in contrast to the mail that the USPS delivers to my door. I don't ask for most of it and I have to sort through it and throw all the bullshit away.
I guess what I mean is that most internet traffic is based on consent of some kind. While paper mail is not.
I might not specifically request any particular bits but the majority don't show up unannounced and unwanted.
> there's plenty of application protocols which involve "pushing" data over TCP/IP transpor
Right but by the time a TCP link is established you've already setup a session which can't be done unilaterally. That is akin to consent and quite possibly a request. You can't be "data-raped" because if you decline the TCP session that's the end of it. Someone can send a billion request to initiate TCP sessions but they can't initiate without your (or your computer's) consent.
Given the router/firewalls that basically everyone has between their modems and their computers it's getting very, very difficult to send data to residential ISP customers without their express or implied consent.
TCP quite literally is NOT UDP. You can do TCP over IP and you can do UDP over IP.
But under NO circumstances could you claim to do UDP over TCP/IP.
Were Netflix performing this "data rape" of Verizon's network it would be huge news as a DoS/DDoS attack. But that's not what we're hearing.
CAN people perform attacks? Yes absolutely and it requires no consent. But without some kind of consent it's regarded as an attack rather than as legitimate network traffic.
It doesn't take too much away from your point, but people do "push" data outside the context of an attack. IP is not a request/response protocol like HTTP.
Agreed, it's technically possible. In practice it almost never happens. And even when it does it might TECHNICALLY be a push but from a very high level perspective it might still be a pull.
As an example I use a website (like Facebook or something) which does live auto-updating. Under the hood it's something like WebSockets or SocketIO and the server does send datagrams to me without me specifically requesting those individual data packets. I couldn't request them, I didn't even know about them until they arrived!
But while that technically is "push" from a high level it's still "pull" in that Facebook (or whoever) doesn't just start sending them to me for giggles. It doesn't start happening until I visit their site, and it stops happening fairly shortly after I navigate away from it. Their computers don't unilaterally send packets to me. I request them by continuing to execute the javascript that their servers sent to me when I loaded the page. Once I stop executing it, the connection is closed and the messages stop.
Downvotes? The person I responded to said you can't push data, you can very well push data without the other end initiating it. I specifically mentioned that you can do it with an HTTP POST (using curl as the UA).
In order for you to "push" as you're calling it, the server that you're "pushing" data to has to ACCEPT your connection attempt.
If it doesn't accept the connection your push goes nowhere. Imagine the webserver is down or the port is blocked. In that case the TCP session doesn't get initiated and you're unable to send the data.
EDIT: Considering that most consumer ISPs don't allow their customers to run servers (Google Fiber is an exception) it would be very very difficult for someone to successfully "push" data to consumers. And that's only if I'm willing to accept your definition of push, which I'm not.
It would only have to accept it with TCP. I could do an equivalent thing with UDP and just send data anywhere I want. (Remember the poster said the internet and not HTTP.)
Also, that has nothing to do with the point that "pushing isn't a thing on the internet" is a poor way of describing the problem being described, especially since the poster said the internet and not consumer's using their ISP (which still isn't strictly true, but is at least closer to the idea of the asymmetry in most consumer and business interent connections).
That doesn't negate the point that consumers download much more data than they upload.
When I say something incorrect, I expect to be corrected. I attempted to correct something that was factually wrong and don't believe I was mean while doing so.
Yes it is technically feasible to perform a push-type operation and send data to someone without their consent. But that's generally regarded as an attack, rather than legitimate network traffic.
I don't have any hard numbers but I would guess that at least 90% and very likely 95% or 99% of all internet traffic is pull-style.
Transferring data between two parties on the internet for any reason usually goes hand in hand with one person wanting to receive some data and another wanting to send it. If the sender doesn't want to send, no transfer takes place. If the receiver doesn't want the data, it's regarded as an attack.
I am having a hard time coming up with legitimate situations where someone sends data to my computer without my permission (express or implied) and that doesn't seem like some kind of attack.
But I'm also open (at least I'd like to believe I am) to someone cooking up a situation that proves me wrong.
This is just pedantry completely ignoring the main point - of course, each tcp/ip connection sends bytes in both directions. But the connection to a residential customers (Netflix users) are very asymmetric - they are almost always initiated by user's initiative and have users request a lot of data flowing in their direction, while almost none flows in the opposite direction. It's not like Netflix just decides to send me a gigabyte of a movie. It's also not like we exchange gigabytes routinely in both directions. It's me asking to get the movie and paying for the privilege of getting it. The fact that technically a tiny stream of bytes also gets in other direction is irrelevant here.
I was being pedantic. If you'd note elsewhere I explain to others the asymmetric nature of consumer ISPs, so I'm not obtuse to the issue. I just believe that saying untrue things is unhelpful to the problem. I want to be called out when I'm wrong; I hope others do to. I don't believe I was mean while doing so.
Also, nothing I said despite it being asymmetric, just that his point about pushing and pulling isn't the best way of describing the issue.
> If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
Not always. Backup is popular today - as I look at my simple home utilization I push about 30% of the ingress up to "somewhere".
> Double billing for a single service is a neat trick if you can pull it off.
This is how every communications network has been ideally built from the beginning and this "trick" has been played out time and time again starting starting with the PSTN. It's not uncommon, but is also isn't ideal - unless you are the carrier.
>Not always. Backup is popular today - as I look at my simple home utilization I push about 30% of the ingress up to "somewhere".
You're equivocating on 'push' and 'pull'. Under OPs definition, a 'pull' is a customer's valid usage of the service, while a 'push' is something like unwanted or unrequested data transfer from a 3rd party. This definition is a little muddled when you talk about uploads but it should be easily understood. You uploading your backup data is you merely using the service you paid for, even though you're 'pushing' data somewhere.
Furthermore, it's not as though the backup company doesn't want the data and you're trying to cram it down their throats anyhow. Are you using more upload that download while doing the backup? Yes. Is the backup company's server sending ACKs to let your client know to continue sending data? Also yes.
Balance-flow contracts are only used when each network is acting as a gateway where the end-point of the traffic is NOT IN either of the parties involved in the contract.
For example, suppose there are 5 separate networks, run by different groups: A, B, C, D, and E. C connects to A,B and D, and D connects to C and E. In this case, the only way networks A and B can connect to D or E is through the connection between C and D.
Now, if in this arrangement, the D or E networks are sending/requesting a lot of traffic to/from networks A and B, while C is rarely requesting data from D or E, then network C might have a fair argument to make that D is abusing their balanced-flow peering agreement. Basically, they are being asked to be a middle-man for all the traffic going between networks that are NOT theirs. They might ask for compensation for doing this job (carrying traffic between two other networks).
However, this is NOT what Verizon does. All of the traffic between level 3 and the Verizon network is bound FOR A VERIZON CUSTOMER. They are not a middle-man, they are the end destination. Now, if Level-3 is sending data over those congested level-3/verizon gateways that is bound for, say, AT&T's customers, then they would have an issue and could fairly demand payment.
"That is not the case, however." is indeed correct. The case is that Verizon does indeed act as a peer for business connections. Verizon is not just a consumer based end point on the internet. This is the crux of it all.
Verizon is charging customers for access to internet content and they're bringing that content in over a peer balanced connection with agreements in place, intended for use in an actual peering environment, by business costumers of Verizon who are paying to provide content in the same way that Netflix pays Level3 to provide their content over their peer connections.
The conflict here is that Netflix is paying to provide the content, Verizon's consumers are paying to get access to the content. That creates a connection chain that is unbalanced in some way. Verizon just wants that imbalance to be double payment for them and nobody else.
I'm not sure why Level3 has to pay anything. Let's imagine there's only 4 entities in the world - User, Verizon, Level3 and Netflix. User buys internet access from Verizon and pays. Netflix hosts servers at Level 3. Level 3 has connection to Verizon. User pays Verizon to be able to access Netflix servers. Netflix pays Level 3 to deliver content up to Verizon. Why when the data crosses between Level3 and Verizon Level3 should pay anything? Verizon has been already paid by the user to deliver the data. That's like a taxi driver demanding payment from the restaurant since he brings them clients. Clients already paid for being brought to the restaurant, that's the whole reason they hired taxi in the first place, and demanding payment from the restaurant sounds like a racket to me. And a fraud towards the users which expect the provider to perform their obligations, not to use them as bargaining chip to extort other businesses.
If there were only 4 entities, your description would make perfect sense. It's a waterfall model, where the data only falls from Netflix to Level3 to Verizon to User.
But the Internet is a big hierarchy where things flow downhill. Every node can talk with every other node. Historically the networks would trade data freely if they were both doing an equal amount, but even 15 years ago I heard sober network engineers worrying that in the future there might be very unbalanced connections and what would be the fair way to deal with that.
NB: In the end I think Verizon is being purposefully difficult to their customers, but they are not completely unreasonable to think that having a lot of data sent into their networks [1] is just something they should STFU about.
[1] yes, even if you say "data requested by their customers" over and over again
The shouldn't shut up about data sent to their networks. They should get paid for it. And they already are! The only reason for consumers to buy services from Verizon (I mean private consumers not running server farms) is to have data sent through Verizon networks and get the data to the users, and of course it is wildly asymmetric - everybody knows it is. It's not some externality that Verizon must suffer - it's the primary and sole reason why users subscribe to Verizon services and pay them money! If Verizon is unable to provide this service for the money they change, they should charge more or change how they do it, but what I completely don't get is how it is taken as if their direct contractual obligations - taking data from whenever it is and getting it into my home (provided I am a Verizon customer) - is presented as something extra that Verizon must unexpectedly do for free.
> (in this case, since Level3 is pushing more data, they would be the ones paying).
That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers.
All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern.
Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).
it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this)
Not impossible though. I'd bet money that equal in/out contracts would be cheaper- maybe they just hoped no one would notice, or didn't anticipate the impact of Netflix on already-existing contracts?
> maybe they just hoped no one would notice, or didn't anticipate the impact of Netflix on already-existing contracts?
It's not just Netflix - the entire Internet works this way. End-users send a request, and servers send a response. There are very few cases in which the response is not larger than the request. It doesn't matter what content you're serving - the raw traffic delivered to consumers is going to be greater than the raw traffic from them[0].
Saying that they didn't know that the traffic would be shaped this way is saying that Level 3 doesn't understand the way the Internet functions on a technical level, which I would say is impossible.
[0] This is (sort of) why residential Internet speeds are usually quoted asymmetrically (e.g. 20 down/5 up). Consumers - well - consume more traffic than they produce, so networks are already optimized for delivering more traffic to consumers than they themselves generate. That said, all of the traffic is initiated by consumers; the traffic Netflix sense is only in response to the explicit request by a paying Verizon customer.
I guess the idea is that backbone providers will each have a similar number of ISPs (and content networks) to deliver traffic from and to. That's clearly not the case, especially with Verizon providing both a backbone and an ISP, and level 3 providing mostly content delivery. Maybe both should work on diversifying the clients that they serve.
You seem to have segmented ISPs in your head. As if there were "Hosting" ISPs and "Client" ISPs. Level 3 provides service to customers of both varieties. You can't jsut look at Level 3 and say, "Well they are a Hosting ISP so they send more than they receive".
> I'd bet money that equal in/out contracts would be cheaper- maybe they just hoped no one would notice, or didn't anticipate the impact of Netflix on already-existing contracts?
You'd actually bet money, on the idea that Level 3—a major internet company—didn't understand a fact about internet traffic that's blatantly obvious to everybody? What?
>Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows
Verizon's argument would be a lot stronger if they didn't attempt to say that Netflix chose balanced flow contracts. They are saying that Netflix went shopping for backbone providers and chose the ones that had balanced traffic flow agreements to deliver video across, which is just silly.
Such rhetorical overreaching is a common debating error, and can ruin an otherwise good argument. Verizon may or may not have balanced flow agreements, I don't know what the contracts are, but now I don't trust Verizon to honestly report on them based on their imputation of Netflix's motivations.
This is the exact argument that Verizon is using -- that Level3 is sending "too much" data. Of course, that data is in response to requests sent from Verizon's residential and business customers. So it's not unwanted -- in fact it's the data that their customers are paying Verizon to deliver to them.
It doesn't matter if there is an imbalance in the traffic flow. Verizon is being paid to deliver requested packets over IP. That's all they have to do. Singling out Netflix is pointless. If Netflix were replaced with 1000 video streaming sites with the same aggregate traffic volume who would they blame for all the data their customers are pulling in? Verizon desperately wants to avoid being a humble ISP with no way to pull the strings the way do with their telco service. That's why they're playing this game.
> Clever technique to draw attention away from what is most likely a contractual dispute.
Oh, please. If that's really what's stopping this from being fixed, then I would be hard pressed to find a better illustration of the need for regulating "last mile" providers as public utilities. We're talking about an investment that is not even rounding error in Verizon's accounting, to connect Level 3's network to a Verizon network that, by Verizon's own admission, has plenty of excess capacity that is not being used. So the whole idea that, if the traffic isn't balanced, Verizon is somehow incurring huge extra costs that they need to be compensated for, is obviously bogus, which means their contractual terms are nothing less than an attempt to use their privileged position to extract huge monopoly rents in exchange for no added value whatsoever.
@sheetjs: I'm sorry, but I simply cannot follow the logic in the Verizon argument that you quote. You quote:
"Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows"
But isn't that a misstatement of fact? CDNs like Level3 do nothing but transfer data, usually media files, onto consumer ISP networks when those ISP's customers' request them.
In what dream world are Akamai and Level3 receiving as much data from Verizon's consumer network as they are sending? Why would any ISP expect a link with a CDN to have a balanced traffic flow?
Level 3 is not just a CDN; it's a major backbone. Likewise Verizon is not just DSL & FIOS because Verizon Business is also a major backbone. Apparently Level 3 and Verizon did have balanced traffic for many years.
@wmf, OK, I did not know that background. But I can easily imagine that the AWS infrastructure is geographically diverse enough that Netflix is using Level3 only as a CDN, not as an NTP.
That would be the case when Level3 would like to peer with Verizon on order to send traffic to another peering party's WAN of Verizon.
But it is Verizon customers, who request the traffic and Verizon is not providing the best quality, because they don't want to upgrade their own equipment and the peering with e.g. Level3. In fact, it should be Verizon paying to Level3.
In my (admittedly limited) experience with peering I've never seen a contract. Lots of ad-hoc verbal agreements, and occasionally emails outlining a mutual understanding, but because it so often is obviously beneficial to both parties it is almost always handled in a casual way and decisions are deferred to network engineers for engineering considerations. I've never seen legal get involved. If Verizon brought it to a point where it is a contract dispute then it would be just as much of an indictment- it doesn't change the substance of the post at all.
There's too much confusion, you're right. What Netflix need is some hard numbers. I suggest they take very detailed statistics at the client level and constantly report them back to their servers, in real time. With all that data available, it will surely ~~artificially balance the network usage and solve the legal aspect~~ allow both parties to make informed decisions.
I love this. All of what I presume was happening behind closed doors is finally getting put into the light. If only there were serious options for consumers to act on based on the information that is coming out publicly.
Theres a common theme in this discussion. We're stuck discussing analogies. Instead, we should be reasoning from "first principles" as Musk suggests.
If we do that, we find that physical link speeds of any variety have improved by many many magnitudes over. As this statement suggests, 10GBit is absolute jellybean in the networking world. Yet, last mile bandwidth available to customers has stagnated. Imagine a perfectly paved autobahn, and here we are discussing if we should be able to drive 20 mph on it, and who should pay for the extra damage that will cause to the surface. Incredulous.
There are some regulatory systems, where in a goldilockes moment, these basic truths are recognized and acted upon. Consider the EU drastically cutting roaming charges. What was so blatantly obvious that even the senile, crooked decision makers at the EU couldn't miss it? All the providers already roam over the internet, at basically zero cost for both parties if you consider that speeds are limited by the over-the-air interface at either end. Yet they kept charging customers the price for a specially erected microwave link from 1960.
I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking.
Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.
This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one.
Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon.
And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal.
But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know.
But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.
But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already.
To me it seems like they're abusing monopoly(ish) power.
> Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.
They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.
Absolutely, and you made the point well. I think when companies start butting heads like this, some people don't recognize that it's not just some corporate bank account, but the results of the dollars they've already given for the services they've signed up for. I think that disconnect is responsible for a lot of the lack of outrage around this.
I wonder if Verizon's customers could form a class action lawsuit accusing it of failing in good faith to deliver bandwidth from Netflix due to intentionally congested handover points.
So any disputes with verizon cannot go to court, they have to be arbitrated. Further, there cannot be a class arbitration. So to vindicate your rights against verizon, you'd have to be willing to bring an individual arbitration against Verizon and bear the costs of doing so which will be exponentially more than the value of your claim. In essence, these clauses immunize corporations from liability under antitrust laws, consumer protection laws; employment laws etc. It is a terrible situation created by a Supreme Court that despises class actions.
Verizon has been given immunity from the threat of a class action by virtue of its arbitration clause class action waiver. Same is true comcast, AT&T etc. You can thank the Supreme Court for this.
Aren't such waivers automatically null and void at least in some states? In EU, such disclaimers are simply a waste of ink, but I recall seeing that some USA states had gone that way as well.
Hmm, I've heard of immunity over class action wiretap related issues, but bandwidth delivery too? Any references? "class action immunity verizon" just has too much noise.
Verizon's advantage should be obvious: they also sell TV service which is in direct competition to Netflix. If Netflix works poorly, that's more customers potentially signing up for FiOS TV.
I feel like this is what people often forget in the net neutrality debate. The pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand) and we need to make sure they aren't rigging the race so to speak.
Going beyond that, they typically have a monopoly or duopoly as an ISP, while they're subject to competition as a TV provider.
Much of net neutrality can be reduced to "don't let monopoly ISPs leverage that into a monopoly in other services".
If Verizon didn't offer TV, this wouldn't really be a problem. If there were 17 other broadband ISPs to choose from, this wouldn't really be a problem. But Verizon does offer TV and they're one of two choices for me and most of their other customers, so it's a problem.
It's even stated parenthetically in the blog posting when comparing European and US Level 3 and Verizon peering utilization, the former ~18%, and the latter ~100% "(where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses)"
Furthermore, if Verizon degrades connections to whichever ISPs Netflix uses, then they have leverage to get Netflix to pay Verizon to peer. If they make it so that the only way that Netflix can offer their service is to pay them for it directly, then that is more money for Verizon. Very clear motivation.
It's not stupid at all. By giving into the racket, Netflix demonstrated that it's not a technical issue at all, so that argument no longer flies. It's clear that the ISPs are capable of supporting the service, they just don't want to.
Well not really. They can just claim that without being paid extra to do so, it wouldn't be cost-effective for them or whatever. I mean that's their public argument anyway - that these services cost them too much and they need to pay up.
This is not and has never been a technical problem. It is purely in the business and politics areas.
One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.
I currently have a 6in4 IPv6 tunnel through Hurricane Electric. At least netflix.com seems to load over IPv6. I wonder if my Netflix experience is fine precisely because my traffic appears to go to HE first, then to Netflix.
Edit: in the past I have had similarly good experiences with YouTube problems. They seem to go away as soon as you start accessing YouTube over IPv6.
Sure, set up an SSH tunnel, make it a SOCKS5 proxy (easy enough to do) to a host in the 'cloud', now proxy you traffic through it (like Netflix).
Watch how suddenly you get 'connection reset by peer', look around and discover nobody in your packet path has any cause to reset your traffic. Now do a straight http proxy (varnish works well for this) to the same machine, run it all day night with bits of http traffic. No issues at all.
No go back to creating an SSH tunnel between your machine and the endpoint. Run traffic, note the mysterious 'connection reset by peer' when some thing upstream sends your TCP connection a FIN.
It could be that their network just has a really hard time with encrypted packets but some how I don't think so. I have tried a variety of port numbers.
EDIT: and I went back and verified I still have 'ServerAliveInterval 60' in my config file for all hosts.
When I was on Comcast, I felt like they were doing this to gaming traffic as well. I have no evidence to back it up except anecdotal experience though.
When I was on AT&T U-Verse they did this with torrents. I replaced my router, OS, and modem thinking the problem lay somewhere on my end, and once I ruled all of that out I called up AT&T to complain and call out the fact that it happened only when I torrented.
Their response was that yes, they do disrupt all torrenting traffic, "because it is illegal and they must comply with copyright holders". Nevermind it was various linux distros I was trying to download and evaluate, all torrenting is automatically evil by association.
I had to switch to Cox because they're the only other option in my area. The 250GB data cap on my plan is BS, but at least I can spend that 250GB however I please.
I had problems a while back with any long-running downloads or streaming video on ATT DSL. Basically made some uses impossible - any large remote data that couldn't resume at arbitrary points. Went completely away when I switched to a third-party ISP - using the same ATT-owned DSL lines.
I signed up for a VPN service right at the end, and that cleared everything up for me and completely confirmed they were intentionally shaping and disrupting traffic.
I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.
Verizon probably needs to pay Level 3 for the bandwidth - so apart from the cost of the hardware (cards + cables) there is a possibly large cost which is not mentioned in the Level 3 article.
Verizon probably doesn't need to pay Level 3 for the bandwidth. Typically peering points like this are settlement free. In other words it's a voluntary exchange of traffic and each side of the line keeps 100% of the revenue from its customers using the interchange.
And Verizon claims that peering points are intended to be roughly equal in directional usage. Which this is not. Are there examples of peering agreements where companies don't care that usage isn't balanced and still don't require some sort of net settlement?
Peering links between backbones (like Level 3 and Verizon Business) are supposed to have equal ratios. Peering between content providers and end-user ISPs don't care about ratios.
Right. If they really cared about ratios they wouldn't offer unbalanced service such as 50/15. They have built their network around users predominately consuming content and then complain when there is too much consumption from specific providers that happen to compete with services they also offer.
An end-user ISP like Comcast has a monopoly on access to their customers, so if they're allowed to charge for peering there is no competition to limit how much they charge. This tends to result in the ISP allocating 99% of the surplus created by peering to themselves.
That's very true. The only limit would be losing customer (HA!) because they didn't connect to the right content providers. There's some interesting network effects here that smell like positive externalities, which would explain the underinvestment. But also suggest that there isn't a good solution.
I'd say that wmf addressed it already. Peering points with one side being primarily consumer services are always going to be unequal due to the nature of the end user and the ways in which consumer ISPs provide bandwidth to their customers. Lots of download, and very little upload speed as an example. As such it's nearly impossible for many peering points primarily used by consumer services to use an equal amount of traffic on both directions. Both sides of the agreement know this and it's still usually in both of their best interests to peer settlement free.
This is a good point, but for a different reason: IIRC, Netflix is paying Level3 for connectivity. If Netflix pays Verizon for direct connectivity, Netflix still pays someone for bandwidth, and L3 loses their cut.
and Verizon wrote in their post that they "working" on this:
"we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network."
But "working" could mean: "waiting for Netflix to pay for it".
Netflix already paid Verizon's ransom and now Verizon is building connects to Netflix but such connections take time to install. Level 3 is arguing over old news at this point, but they're trying to set a precedent to prevent other ISPs from trying the same thing.
Let's not forget that those providers that Netflix chooses to use are global edge service providers for high bandwidth streaming content services. Cogent, Level 3, Limelight, etc... Their whole business is providing high capacity back-end network transit which they do well, otherwise they wouldn't be in business for very long. High-five Level 3 for calling B.S. on this!
Funny you'd mention Cogent and L3 together, seeing as L3 has done worse to Cogent (depeering) for the exact same reason Verizon is using (unbalanced traffic).
It doesnt matter what content is coming down the wire, if I pay for internet I expect to be able to access or connect to anything thats not forbidden in the EULA.
With comcast my data cap is 300GB a month, I should be able to use 300GB of whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt matter where it comes from I'm paying for 300GB of data at X speed. How hard is that to get?
Cool! Now tell Comcast you want to download 300GB from my server, which I've got connected on a single 300 baud channel. Will they come give me a free 1GE connection?
Netflix, on the other hand, has paid good money for a very fast and reliable connection all the way to the Level3/Comcast PoP.
For yet another analogy in this thread - if your server was instead sitting in a data center with an open 10GE ethernet port or two, right next to a Comcast router, yes, they should just install the extra NICs and wire the things up without for the price of supplies and installation.
It doesnt work like that because Netflix has already paid level 3 who does most of the heavy lifting. Similar to if you had a 10GE connection at your sever but comcast is still running a 300 baud channel on the connection they have to your server.
OK, so I'll buy a 10G port to another random ISP. And they'll go to the ICX and say "hey Verizon, hit me up with the 10G port". Oh wait, it doesn't work like that. It works with peering arrangements, hmm.
Why should netflix have to pay for last mile when they already pay their backbone provider, who may own most of the wire? We can come up with any scenario or one off situation but still the problem remains.
Should net neutrality go away? And is it fair what Verizon and others are demanding from Netflix/Youtube... Also, who should foot the bill, the backbone providers, the content providers or the last mile providers.
You talk of peering arguments yet you dont seem to back them. If netflix is a level 3 customer and level 3 has a peering argument with Verizon, then whats the problem?
L3's peering contract with Verizon requires roughly equal traffic. Maybe that's dumb, shouldn't be the case, unfair to Verizon customers, etc., but that's literally the issue at the moment.
L3 did the exact same thing to Cogent before, for the same reasons.
One question: There aren't that many large edge providers around. What would Verizon do if Netflix signed a contract with a dozen of the larger ones and routed their streams through whichever had the best connectivity for a given endpoint? If Verizon throttle all of them, they are going to get a lot of collateral - essentially degrading the Internet at large for their customers. Or would they just do packet level filtering then?
I've wondered this as well. I suspect that it would be cost prohibitive for Netflix to have a contract with several edge providers like Level 3. I'm certain that Verizon's price demands to Netflix would be cheaper than this option.
Think about your own ISP. I pay $80 for service, but to get the same bandwidth shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware costs (routers to each ISP).
True. But I would assume that the contract is largely a pay per usage or at least expected usage. So if they just make an estimate of how many Verizon customers they have and buy that block from someone else than Level 3. Then route all that traffic through the other provider, while keep using Level 3 for the majority of their traffic.
Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make the deal for them. These edge providers already have business with each other (peering), so it should be relatively simple for Level 3 to purchase some bandwidth from another large provider and then route some of their packets (e.g. the Netflix ones) through there.
In any case, the edge providers have good reason to collaborate on thwarting Verizon on this, so why don't they?
I think the costs work out differently at terabit scale. Sending a terabit split across two backbones may be the same cost or cheaper than sending it over a single backbone.
Is it just me or is this not at all inconsistent with Verizon's arguments? They've been saying the whole time that this link is meant to be balanced with roughly equal upstream and downstream, and they showed it was not. What am I missing?
You've missed the part where they're trying to double-bill for the same network traffic.
Verizon wants to bill it's customers for the bandwidth that they use. And then for some of that traffic, if it's from Netflix, they also want to bill Netflix.
Verizon is the company which is selling asymmetric internet plans so it should come as no surprise to Verizon that their customers are requesting more data than they're sending.
But they use this asymmetry to suggest that what's happening between Verizon and Level3 isn't business as usual (which it is) and that they should be able to double bill.
Verizon is doing a decent job of spinning but if you really look at what's going on it becomes clear that they're really torturing definitions to make their arguments.
I agree, the fact that they are selling asymmetric internet plans should show that they're clearly aware that the average internet user pulls down far more data than they upload, but (according the Verizon) Netflix's asymmetric traffic is a complete surprise and a burden so Netflix should also pay Verizon for the additional asymmetric traffic being requested by Verizon's customers (who already paid for it).
It's not "additional" traffic if it is within the limits that the Verizon customer paid for (which they did). Verizon just sold something they weren't willing to provide.
I agree to an extent. But there's a point at which Level3 stops becoming a peer and starts becoming a subscriber. That point is somewhere between equal utilization of Verizon's peering point and (near) total upstream into Verizon's network, just as any Verizon subscriber would have. Where is that point?
I'm a Verizon subscriber and I get apoplectic when Netflix stutters, because I know exactly what's happening. But I also don't know what's the exact right model for peering points, and I think net settlement might be a better model.
> I agree to an extent. But there's a point at which Level3 stops becoming a peer and starts becoming a subscriber. That point is somewhere between equal utilization of Verizon's peering point and (near) total upstream into Verizon's network, just as any Verizon subscriber would have. Where is that point?
That's not true. I pay Verizon $X/mo to deliver the internet to me. And most of what I want delivered at 100mbps is netflix. Now instead of Verizon taking the millions (billions?) in revenue and building out what we're paying for, they're trying to charge netflix (for what they already charged me).
I have a website I host from home. Am I entitled to free FIOS because someone else on FIOS wants to access it? That's the question. It gets confused in the peering issue. And settlement free peering is important, even essential to how the internet works. And the FCC should absolutely preserve it.
But the real question here is, is this a peering relationship anymore. And Verizon has a legitimate argument that it is not based on the sustained differences in traffic flows. I don't know what the answer is to this problem, but I'm not convinced by either Level3/Netflix or Verizon.
Personally, if I ran Verizon, I'd be meshing with Google/Netflix/Amazon's networks as much as possible. But I'm not, oh well.
> And Verizon has a legitimate argument that it is not based on the sustained differences in traffic flows
Verizon and comcast have been selling asymmetrical residential links since always. Verizon is advertising 500/100 on their homepage right now. They know damn well that people are pulling way more data than they're pushing, and the interconnect links from level3-to-ISP have always been hugely asymmetrical.
Netflix isn't a Verizon subscriber. Netflix is what Verizon subscribers pay to access.
The problem is that Verizon has apparently been successful in spinning the story that peering between Verizon and Level3 is supposed to be balanced. As others have already said in these comments, that doesn't make a lick of sense. Peering between backbone providers should be roughly balanced, but Verizon isn't acting as a backbone provider, they're acting as a residential provider, and there's no way peering with Verizon will ever be balanced. That's a fundamental consequence of the asymmetric plans Verizon sells and the fact that their customers are expected to download significantly more than they upload. This means that Verizon will absolutely be receiving a lot more traffic into their network than they send out.
Basically, I doubt Verizon actually has balanced traffic with any other provider, except perhaps other residential providers. But they're only making a stink about Level3 because they're trying to use this claim to double-bill.
Note again, Verizon made the intentional choice to service the residential market, and to sell asymmetric plans. They knew going into all of this that they wouldn't have balanced traffic with providers like Level3. And that's perfectly fine, because the only reason they're receiving this much traffic is because their paying customers are requesting it. Verizon has already been paid to receive this incoming traffic.
Also, and this is something I haven't seen anyone really address, balanced peering agreements between backbone providers is typically meant to prevent one provider from routing traffic through a second provider's network, destined for a third last-mile provider. In that scenario, the second provider doesn't gain anything from the traffic, and hasn't been paid for it. That's why the balanced peering agreement exists, to ensure that no provider gets taken advantage of that way; if you one provider wants to route traffic through another provider's network, they have to be prepared to receive just as much traffic.
But that argument doesn't apply to last-mile providers. The traffic isn't being routed through their network to a third destination. The traffic is being delivered to the network because that's it's destination. If the traffic was unsolicited (for example, a DoS), then it's reasonable for the last-mile provider to try to charge the sender for it. But if the traffic was explicitly requested, which is generally the case (and certainly is for Netflix), then the last-mile provider has no justification for charging the traffic sender. The traffic has already been paid for, by the subscriber who requested the traffic.
You make a much better point than I've seen here. Verizon acting both as a backbone and as an ISP. This creates an internal conflict between those businesses.
Just keep in mind that you have already paid Verizon to provide you with "up to X Mbps" of internet. And they theoretically should make a good faith effort to provide you with that amount.
What's happening here is that they are not making the good-faith effort. They want to get paid for the transit from Level3 onto their network and from their network to your front door.
The thing is according to basically all precedent of how the internet works when you pay for internet you're (theoretically) paying for all costs Verizon incurs while providing that service to you.
If they're trying to bill someone else for entry into their network while billing you for the exit then you should be getting a discount to the tune of however much that service provider is paying. But you're not. Which is why people are getting all up in arms.
EDIT:
> Level3 stops becoming a peer and starts becoming a subscriber
Yes, that point is when Level3 is DOWNLOADING more data FROM Verizon customers than UPLOADING to them.
Verizon charges their customers every month for plans where they download more than they upload.
As it stands Verizon customers are downloading 3x as much data FROM Level3 and Level3 is downloading from Verizon.
So by that logic, Verizon should be a Level3 customer and pay Level3 for any imbalances.
To your edit: No, my theory was that, when the businesses aren't servicing each other, as in a mutual transit scenario, then one getting a benefit at the expense of the other is unfair.
Honestly, it's a case of externalities, so we're not going to see much improvement until government either takes over peering or last mile.
I don't think that either is benefitting at the expense of the other. Netflix pays for huge amounts of bandwidth to all the residential ISPs where their customers are located. Verizon customers pay for huge amounts of aggregate bandwidth to Netflix and everywhere else on the internet. If there is more download traffic on Verizon's network than there is upload traffic it's only because that's all Verizon allows their customers to do. Contractually. They sell asymmetric plans. The results are not surprising.
I think Verizon has an internal struggle. They are both a transit provider and an end-user isp. I think their transit role, which made this deal with Level3 for settlement-free peering, and its ISP role, which made deals with end users for high bandwidth, are fighting each other. We'll see if that plays out.
Waaaait, at that point Verizon stops being a peer and starts becoming a subscriber, since they have a bunch of consumers that don't host valuable content within Verizon, but want access to content outside of Verizon.
If that's not a settlement-free peering agreement because of the disbalance, then Verizon should be the one paying part of their users subscription fees to the rest of the internet backbone.
Residential ISPs have pretty much never, in the history of ever, not had asymmetrical traffic. Netflix may have aggravated that, but they did not change that.
What made me laugh the most was when Comcast posted about a peering dispute with Level3[0]
>Q8: Comcast says that Level 3 sends it 5 times the traffic that Comcast sends Level 3. Is that true? If it is true, why shouldn't Level 3 pay for the traffic it sends to Comcast?
Comcast goes on to argue that Level3 pushes 5X more data onto the Comcast network than Comcast pushes onto the Level3 network. What I find ridiculous about the whole claim is that every single one of Comcast's residential internet packages has 5X higher download speeds than upload speed. (e.g. 105Mbps/20Mbps, 50Mbps/10Mbps, 20Mbps/4Mbps)
Comcast provides internet speeds that force customers to download 5x faster than they can upload and somehow Comcast is surprised that their customers are pulling down 5X more data then they're pushing to Level3?
Following Verizon's argument to its logical conclusion, if the link is unbalanced then Verizon needs to buy transit from Level 3 to make up the difference.
Thinking only of Verizon the consumer ISP, in what universe are consumers creating as much bandwidth as they consume. Upload speeds for consumer internet are what? 20% 30% the download speed. Verizon the consumer ISP would never generate as much data as they pull.
Netflix did offer to have their client generate random upstream traffic to balance the load. Which netflix says was met with silence in the meeting where they proposed it. It has some amusing implications where netflix saturates verizon's network to pay less.
I think a natural solution to this is to have Netflix charge all Verizon customers a $5/mo surcharge to pay for 4 more 10gbps ports.
But after reading this... am I the only one that thinks peering agreements are done bass ackwards? If Verizon's customers are _requesting_ Netflix access, shouldn't Verizon pay Level3, not the other way around?
This is the entire net neutrality debate in one simple blog post. Verizon is selling internet access to their users but not actually providing real access!
Point of information about Verizon as a purely residential carrier: Verizon owns MCI/Worldcom and Terremark. They're not just a residential carrier. I don't have any idea what the ratio of residential to business traffic is, of course, but I think Verizon is a real Tier 1 carrier. Happy to have someone with more definite data chime in if I'm wrong.
Verizon might technically be one company but in reality there are several business units which are actually separate companies in all but name.
When you buy business backhaul from Verizon you get guaranteed bandwidth provisioned on their network from point A to B. They don't over provision because they can't: contractually. If they don't meet their SLA the customer is out of the contract and can go elsewhere.
On the residential side things are very different. We have no SLA other than "pfft whatever we'll roll a truck next Thursday I guess"
Verizon might share long distance backhaul between residential and commercial sides of the company but if so it's done with strict quotas to ensure that the commercial side of things isn't impacted. It's not as though the core network division lets everyone have a free-for-all on the available bandwidth through the core.
If I had to guess I would venture that the consumer group either runs largely or entirely separate equipment. They might share fibers but I'm not all that confident about that. They're very, very close to entirely separate businesses.
How did people think this wouldn't happen when ISP's are allowed to be content providers, too. Of course Verizon wants to choke Netflix and promote their own service instead. Letting cable and content providers be ISPs was always a bad idea.
It's working for me now as well. When I posted the cached version the page on Level3 blog was giving me a 404 although the post was on the "Recent posts" list.
I'm not getting why I'm being down voted for posting the cached version when I was getting the 404.
i recently gave up my netflix and amazon prime because i was tired of streaming hanging up in the middle of an episode/movie. it was just annoying and embarrassing if i had guests.
now i just torrent everything so i can watch uninterrupted when i want. I consider i am still legal since i'm paying for both subscriptions yet and they pay the content. if not, not my problem.
but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?
as i just showed, they are the reason lots of people are going back to torrents that do not make money to the studios like streaming services does. Why do they rather sue john does instead?
"but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?"
Because it presupposes that whether or not in fact, streaming via Netflix is profitable or not, the studios didn't have to be dragged kicking and screaming to allow it at all.
I've always been suspect of cable and wireless internet connections. Anything that shares bandwidth with all subscribers is going to get maxed out if a large number of customers start streaming video. Is it possible that Verison (and Comcast and etc...) know that opening up the gate to the outside world will crush their network under the increased load? I like the milk-our-monopoly-on-the-last-mile theory as much as the next guy, but is it even possible for the technology used to deliver the necessary bandwidth if they open up the pipe?
Verizon could have legitimately argued artificially throttling at the Level 3 connection is necessary to protect their network. But they didn't say that at all. They basically bragged that their own network is running well under capacity. The insinuation is that Verizon is ready to handle all the bandwidth that anybody can send, but that Level 3 was over capacity.
But, assuming they really did need to throttle traffic in order to protect their own network. For one it wouldn't make sense to control your throttling by running your routers at 100% capacity. But also any throttling should be equal among all of their peers and not selectively chosen based on Verizon's business interests.
Net Neutrality isn't about usage, it's about destinations.
Even with Net Neutrality it's fully within an ISPs right to say X GB/mo for $Y and throttle or block over-use. The problem is an ISP deciding that, regardless of what is going on in the network, regardless of what a particular user has used or is trying to use, traffic to Netflix will be slower than traffic to Vudu.
Verizon, Comcast, et al have plenty of available tools to prevent congestion without violating Net Neutrality.
They're intentionally trying to confuse the issue with usage, because it's an easier PR sell.
Oversubscribing lines has been one of the ways to keep ISPs profitable for years. Streaming video absolutely cuts into the ability to do this of course but I doubt it's the primary issue here. Verizon has every incentive to not add more bandwidth to their peering points with Level 3 and try to get more money from Netflix directly.
nope. if you look carefully, the math doesn't work out. verizon will be fine. They want to charge netflix and have people pay for tv programming. The current legal way to do this is to claim they don't have the capacity (between what and where is unspecified, and by refusing to add it - even when it's 5 minutes and free).
My point is was that spectrum doesn't scale. There's X gigabits per second available over the air, and a similar X over a cable. For cable companies they can push equipment closer to the home and have fewer subscribers on the same line. I suppose Verizon could do the same with more towers, but imagine doubling the number of towers to support double the bandwidth (or something like that). As people have pointed out though, the congestion is specific to connections that carry netflix.
There's a limit for mobile bandwidth, where also the number of towers won't help; but cable is different - sure, at one point they have to put proper fiber to the home (and it's economically feasible to do so, if there's actual competition instead of monopolistic ability to charge huge prices for weak service), but after that you can have as much bandwidth as your routers can service.
Summary: The bottleneck is the the interconnection point. Netflix is saturating the interconnection point, and Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic".
But Level 3 claims it would be very inexpensive for Verizon to increase capacity there. Just a few port cards and a few thousand bucks for a major city like L.A. They even offered to pay Verizon's costs.
Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack.
This is Verizon wanting to double-bill for traffic. Plain and simple.
> Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic".
That's Verizon's job. To go out and get by any and all means necessary enough bandwidth to satisfy their customers, namely the residential customers who seem intent on using Netflix. At no point do they have a moral imperative to hold their customers hostage to extort money out of someone else that their customers are trying to access. It's already been paid for!
> Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack.
All of your comments keep ignoring or misunderstanding what peering is. It is supposed to be roughly equal data traffic for both sides. Once it is lopsided in one direction, it is no longer peering.
It doesn't matter who requested what, the fact that L3 traffic is consuming a majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix that.
It really doesn't matter what the company name is.
At the peering point the arrangement is no longer mutually beneficial for both parties due to one side of the peering arrangement now using more than its fair share or traffic. This is what Verizon has issues with.
You continue to miss the point that L3 doesn't consume bandwidth. The people that consume bandwidth are Verizon's customers. And Verizon is failing to procure sufficient peering to satisfy their customer's demands.
You can start talking about peering and Tier 1 ISPs but Verizon Residential is not the same company as Verizon Business. Verizon Business is a Tier 1 ISP, Residential is not.
L3 can't consume any bandwidth on Verizon's network because their customers don't unilaterally push data to Verizon customers. If there is any traffic from L3 to Verizon it's because Verizon's customers have requested it. That means Verizon should make a good-faith effort to acquire sufficient bandwidth to sate their customers requests all the way up to their theoretical contractual obligations. Only once the bottleneck is COMPLETELY out of Verizon's control do they have any room to suggest that "well our service is only up to X Mbps and in this case it's out of our hands"
In this case they have the power to fix the problem and they are choosing not to, to the detriment of their customers. It's shameful.
EDIT: If the link were 100% saturated in the other direction then you could potentially make the argument that L3 is causing the problems. In that case their customers are requesting so much data from Verizon's customers that the link gets congested. And in that case it would be paramount for L3 to figure out a way to get Verizon to upgrade their side of the peering agreement. But that's not the situation here.
I'll reiterate again but I don't think it will make a difference.
There is a peering ___location. At this ___location traffic is supposed to be roughly equal in both directions. At one time it was, and both parties were happy. It no longer is the case. L3 is now consuming >50% of the bandwidth. Verizon states that since they are using more than their fair share they should pay cash to upgrade the network. It doesn't matter who requested what, or from where.
Also your edit is correct. If 90% of the traffic at the peering point was outflow from the Verizon network, then they should be paying to upgrade the bandwidth.
I guess what I'm trying to do is understand the rationale behind your argument of "sender pays" and who is "consuming bandwidth" because it's not clear to me how things should be measured.
I will sketch out my argument as follows:
1. Data transfer requires mutual consent or else it either a) doesn't happen or b) is considered a network attack
2. On the internet today the majority of transfer flows are asymmetric; regular users download more than they upload. I don't know if this is the natural way of the universe or it happens this way because of the way residential broadband is structured i.e. more download capability than upload
3. People who have more outflow than inflow pay to be connected to the internet (i.e. servers)
4. People who have more inflow than outflow pay to be connected to the internet (i.e. regular users)
5. Somewhere this traffic has to meet in the middle and transition from a network where the sender is paying to the network where the receiver is paying
6. This is typically called a peering point and the transit here is usually settlement-free not because it's usually balanced (though in many cases it is) but because it's the point at which the sender-pays network meets up with the receiver-pays network. In other words the sender pays for the network transit from their servers all the way up to the peering point and the receiver pays for the network transit from the peering point onwards to their house.
7. When looking at the problem this way it's hard for me to determine who is "using" more network bandwidth because again, data is only transferred by the consent of parties on both the sender-pays and receiver-pays network
I understand the argument that Verizon is a Tier 1 ISP and thus deserves to be paid for transit. The Tier 1 ISP is actually Verizon Business a largely separate division of Verizon from Verizon Residential. But Level 3 is also a Tier 1 ISP and thus by that logic also deserves to be paid for transit. http://en.wikipedia.org/wiki/Tier_1_network#List_of_tier_1_n...
I also understand the argument that "all peering needs to be roughly equal or else someone needs to pay" but again when company A has customers who send more than they receive and company B has customers who receive more than they send I can't understand why A and B don't agree to let the ratios move away from balanced. I understand historically the ratios were close to even but when they're both selling asymmetric service to their customers why insist on symmetric peering?
EDIT:
From Wikipedia "peering" is sometimes called "sender keeps"
"In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network. The pure definition of peering is settlement-free, "bill-and-keep," or "sender keeps all," meaning that neither party pays the other in association with the exchange of traffic; instead, each derives and retains revenue from its own customers."
I don't believe that I am. There is no natural law of the universe (like gravity or electromagnetism) that says that peering MUST be symmetric or roughly symmetric.
A symmetric or roughly symmetric peering arrangement naturally arises when two companies which have both traffic producers and consumers on their network interconnect their networks at a peering point. But if one network operator doesn't have the right mix of producers and consumers on their own network the balance can get pushed in one direction or the other.
From their website:
"The Peering Coordinator Community put on a debate on the rationality of peering ratios as a peering discriminator at NANOG 35 in Los Angeles. During that debate, and during the subsequent informal debates afterwards, the consensus was that this metric was neither technically sound nor business rational."
Peering isn't about getting exactly identical amounts of traffic both ways. A lot of times that is the outcome, but it is NOT required.
Peering is about the point at which two companies interconnect in such a way as to keep their paying customers happy without incurring additional expenses.
Please read (and re-read if necessary) that page and look at the diagrams and make sure that you understand them before you continue to comment on this topic. Your comments thus far indicate that you don't really understand how networks work. It sounds like you've heard "peering" before and kinda understand what it means. And this dispute between Level3 and Verizon doesn't fit into the neat and tidy little "peering is symmetric" box that a naive understanding of peering would yield. I am sorry if this sounds condescending but you're making a lot of posts with no real bearing on what's actually happening or why.
> Peering is about the point at which two companies interconnect in such a way as to keep their paying customers happy without incurring additional expenses.
This summarizes it. Both sides at one time were happy, now one side is not due to uneven network usage (which they consider an additional expense).
That is all.
Unfortunately this whole thread will go round and round forever because many on HN believe no matter what Verizon / Comcast, etc do is automatically evil, and nothing will change that.
> This summarizes it. Both sides at one time were happy, now one side is not due to uneven network usage (which they consider an additional expense).
"which they consider an additional expense"
Verizon can CONSIDER it an additional expense all they like. But it is a cost of doing business that they took upon themselves as a result of the service which they have sold to their customers.
Verizon has only incurred this cost because they advertise and sell asymmetric internet connections to their customers. Which their customers then expect to be able to "collect" on, so to speak.
"keep their paying customers happy"
In this case we can see that Verizon is not acting to keep their paying customers happy. Level3 was never a paying customer of Verizon. Verizon Residential subscribers have been and continue to be customers of Verizon.
What Verizon is arguing is that Level3 should become a paying customer of Verizon in order for Verizon to be willing to spend the money required to keep Verizon's already paying customers happy.
You might call that "double charging" or "abusing monopoly power" or perhaps even "holding their own customers hostage" but I can't see how you can call it "perfectly rational".
I don't think that Comcast or Verizon are inherently evil. I've had good experiences doing business with both on the commercial side.
But when they deliberately throttle their customers network connections at peering points for the sake of extracting additional money from content providers because they have a monopoly over their residential customers I can't help but get at least a little bit upset.
If there were viable alternative choices for broadband, let them do whatever they want. I can always switch away from them to someone who provides me with better service. But when they actively lobby to prevent municipalities from constructing their own broadband networks (which would provide competition) and when they actively lobby to get the FCC to change the rules to favor themselves over consumers BY LAW I also have a problem.
Introduce enough competition into the marketplace and I don't care what any one particular player does. But lacking that competition I can't help but pass judgement on those who do not play fair.
Do residential ISPs really have roughly balanced peering agreements? That doesn't compute to me.. aren't most residential connections super asymmetrical?
Yes, residential connections are supper asymmetrical both in capabilities and their usage, but residential ISPs don't have "roughly balanced peering agreements" because they don't have peering agreements at all.
If most of your traffic is residential, then you're not a peer, then you're a subscriber - residental ISPs have to pay others for their Internet access. Verizon has also a valid transit&peering infrastructure, but if we would be looking at their residential business separately, then it would have to pay for the customer<>Netflix traffic on their interconnections.
What's up with the downvote? Is that not a fair summary?
The haggling over semantics here doesn't really change the substance. Ok, "Netflix" isn't saturating the interconnection.. "customers using Netflix" are saturating the interconnection. And it would still reportedly cost Verizon very little to fix it.
I think that points the finger pretty strongly at Verizon.
Greed! It would not stop here. Today it is Netflix and youtube, tomorrow it will be image heavy sites. Then they will go after google, mail provider and bank sites etc.
Great post. Frankly Level3 should be pissed off. Verizon essentially called them out for providing crap service when in fact they are the ones responsible.
What would happen if Netflix stopped allowing access to its service to Verizon customers in areas where there's another provider? Netflix could argue that it doesn't want to provide an inferior product if there's a choice, and maybe leave a number on the homepage for the cable company in your area.
I wonder if Netflix or Verizon would lose more customers...
Then a) Netflix becomes the bad guy, b) they lose money from those customers leaving, and c) they no longer have the opportunity to influence those people.
They should continue to allow access, but they should also keep saying when the problem is with the network provider. Customers love Netflix (NPS of 54) more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.
The claim from Verizon that still stands after this blog post is the fairness argument to peering. If you peer between symmetric networks, it's typically free. That's not the case here, so Verizon wants to be paid for the asymmetry.
Consumer internet providers will always have assymmetry because people generally are consumers of content, not creators or distributors. Remember that standard home internet connections have assymmetric download/upload speeds and also ban consumers from running hosting via their consumer grade internet connection.
The issue is that Verizon is both a transit provider and a last mile ISP.
For transit providers, asymmetry = Paid.
For local ISPs, they are typically willing to asymmetrically peer because they'd otherwise have to pay transit.
Since verizon is a T1 transit provider, it doesn't want to give it's service away for free.
I think the fair answer in this dispute depends on whether L3 and Cogent are forcing Verizon to provide transit for free or are they just delivering the data the local network for the customer (like a local ISP peering arrangement).
I'm not sure which is going on. If L3 is just dumping Netflix packets on the nearest Verizon peering ___location, well then Verizon is providing them a transit service. If L3 carries the data all the way to the local network, then it's not transit at all.
I'm not sure what is actually happening. I'd imagine a mix of both.
Level 3 is a tier 1 provider so in theory they should be peering for "free" with other Tier 1 providers such as Verizon.
From Wikipedia:
"However, the most common definition of a tier 1 network is a network that can reach every other network on the Internet without purchasing IP transit or paying settlements."
Not really, for the consumer ISP side of its business Verizon absorbs more traffic than it produces by definition: consumers don't typically produce much data outside of p2p.
Furthermore, Netflix has offered putting server inside Verizon's network which makes this argument moot.
One solution would be for Netflix to modify their clients to send useless and quickly discarded UDP packets back to balance out the traffic. Verizon then has tons of data to push back to Level3 and everyone's happy.
That wouldn't work though. Verizon sells unbalanced services with larger download capacity than upload capacity. So it's simply not likely/possible that their customers can send matching data for what they are requesting. Additionally, while this might not negatively affect their FIOS customers their DSL customers are on asynchronous connections meaning that the uploaded data will negatively impact their ability to receive data at the same time in effect degrading their ability to stream and the quality of the video they can receive.
What DSL cusotmers have are asymmetric connections, not half-duplex. Even on ADSL, you can watch a HD video stream while uploading stuff. The only way the upload matters is if it prevents timely delivery of ACKs for the video being downloaded, and that's only a problem on the lowest speed tiers and when you don't have decent QoS.
Thanks for catching that error, as I typed out asynchronous I was saying to myself "this isn't right, the a is in the dang acronym" and then I was too lazy to double check.
So the client should just keep sending useless UDP traffic in the background even when the customer is not watching anything. I'm sure that can be evened out somehow :)
You could get it much closer to balanced. A good quality Netflix stream is 3Mbps down and probably only a few dozen kbps up. You just have to send some occasional bookkeeping data.
What if Netflix got that number to go up from say 30kbps to 500kbps? Still only 10% of upload speed link utilization on a 50/5 Mbps customer line.
Even if the ratio only went from 3:1 to 3:2 suddenly you're much, much closer to balanced. Takes a lot of wind out of their sails. Instead of "ZOMG THREE TIMES AS MUCH!" it's "FUCK IT'S 50% MORE!" which doesn't sound nearly as good.
The Internet is a crucial part of our infrastructure and a important backbone of our economy. If there is any indication that someone like Verizon in this case is tempering with it intentionally, they should be charged with terrorism. This is no different than someone going out and sabotaging Hoover damn on purpose. In fact it's worth!
That's what I would call a mortal wound to Verizon's arguments.