Hacker News new | past | comments | ask | show | jobs | submit login
Netflix Agrees to Pay Comcast to End Web Traffic Jam (wsj.com)
487 points by dctoedt on Feb 23, 2014 | hide | past | favorite | 337 comments



And so the precedent is set. This is the direction I see things going:

For $70 a month from Comcast you can get Amazon, Google, and Facebook. For only $10 more a month you can add Netflix and Spotify. Want Pandora? Sorry, you'll have to switch to AT&T for that.

The broadband providers are used to this type of packaged service given their history as cable television providers. It's how they make their money beyond providing simple connectivity. Their future expansion and profits are linked to their ability to create exclusivity and extract revenue based on targeted demand.

I don't see any other way around this. Absent regulations prohibiting packet discrimination, I'm not sure I would do any different if I were in Comcast's shoes.


This is exactly the opposite of what you are talking about - there is no preference being provided for Netflix traffic. There is no charge going to the customer for access to Netflix.

This is simply the case of a provider, discovering that Cogent provides poor transit services, realizes that you actually have to pay in order to get good Internet peering. This has always been the case on the Internet.

As long as Comcast treats all traffic the same regardless of source, destination, packet type, etc.. I'm fine with their behavior here.


According to the article, "the average speeds of Netflix's prime-time streams to Comcast subscribers had dropped 27% since October"

I guess the question is, did all other traffic see the same drop? And what changes in their network would cause this?


Exactly - was it the case that ALL cogent traffic being blocked - from everything that I've read, it was. Cogent's circuits were just overloaded. They had sold transit to Netflix that they were unable to provide.

Netflix customers, in this situation, can point their fingers at Netflix (who, in turn, can point them at Cogent).

If Cogent provided 1 Gigabit of transit to Netflix, but didn't have circuits with which to connect Netflix to their customers, well, It will be interesting to see what kind of Liquidated Damages Netflix wrote into their contract with Cogent. Maybe it will be enough to pay their peering fees with Comcast.


Why should Comcast provide transit at a a reasonable price to Cogent when Cogent hosts data that competes with Comcast's own services? Their uneven traffic argument is a relic of the telephone era -- Comcast's customers pay them for 20 Mbps of bandwidth to services they want, but what they get is just 20 Mbps to speedtest.net. It should be on Comcast to ensure their network can get their customers the data they want.


> It should be on Comcast to ensure their network can get their customers the data they want.

In your opinion why doesn't the same responsibility apply to Cogent?


Sonic and Google Fiber manage it fine. This is clearly a case of Comcast trying to get paid from both ends and screwing their customer with poor access to the services they want in the meantime, which conveniently nudges their customers to their own competing services.


>It should be on Comcast to ensure their network can get their customers the data they want.

I agree in spirit, but how do you make the distinction? If Cogent has poor connectivity to the wider Internet, is it on Comcast to fix that?


Cogent has been trying to upgrade their connection to Comcast and Comcast has been stalling working with Cogent for months. It's pretty clear cut in this case. A rule when the CDN and the ISP already have a direct connection is not overly unreasonable. Comcast customers are requesting a large amount of data from Cogent, Cogent tries to work with Comcast to improve the existing connection, Comcast refuses.


Verizon customers have had an ongoing similar issue, also with Cogent peering. As I've stated in comments past, I had to threaten to cancel our business FIOS plan because, with 50% packet loss, the route to one of our colocation facilities was effectively unusable.


Actually yes, anecdotally. I have a dedicated server in a DC and Comcast at home, and my SCP speeds went down over the last few months to the extent that I had to move DCs to another with no Cogent.

Tata is also spectacularly bad with Comcast.


What if, say, ISP1 and ISP2 both have shitty performance of netflix and then netflix pay for ISP1 to make it better. They do not pay ISP2. So, essentially netflix is available only for ISP1 and not ISP2. That sounds like that the parent comment is saying and what could happen here.


Why would Netflix want to do that?


Because maybe ISP2 wants more money, or Netflix can't afford to pay everyone off.


ISP2 will see their netflix-using customers leaving for ISP1 in droves if that's the case.


What if ISP2's customers don't have access to ISP1? Given the state of our broadband infrastructure that's MORE than likely.


The deal was a paid-peering deal, which would likely drive down Netflix costs. We don't have any information on whether net neutrality was harmed. Hopefully netflix will provide information on this.

Peering is a practice where two sides agree to maintain their side of the infrastructure and not charge for metered bandwidth.

https://en.wikipedia.org/wiki/Peering


Why do you think it would drive down Netflix costs?

In old world there were a dozen of "tier1" providers who would all peer with each other more or less settlement free. It is conceivable that one of these providers (I think it was Cogent in this case) was offering "low rates" to take Netflix traffic and deliver that traffic to Comcast over their settlement free peering. As traffic flow between Cogent and Comcast was increasing, Comcast refused to upgrade the interconnect to accommodate the flow. Then: links get hot, traffic gets dropped, customers complain, Netflix+Cogent asks Comcast to behave, Comcast offers to unblock the pipes via direct Netflix<->Comcast peering for a "fee".

Now I don't know whether that "low rate" that Cogent was charging Netflix was lower than a "fee" that Comcast is charging. It might be that you are right and the fee overall reduced Netflix costs. What I do know however, is that Comcast is a monopoly and if unregulated, they can start extracting higher and higher rent from Netflix (or any other content provider for that sake).


Peering agreements existed in the form they did because of the underlying assumption that traffic would be roughly symmetric. Services like Netflix that send massive amounts of traffic in a single direction violate this assumption.

This is one of the underreported issues in the whole net neutrality debate. In the late '90s I worked in the WAN engineering group doing capacity planning at a big regional CLEC, which actually was a Tier 1 backbone provider. It was widely assumed in the industry that the flat rate pricing model -- pay $X for Y "guaranteed" bandwidth -- was living on borrowed time and would be replaced by actual usage-based pricing, paying for packets like you pay for gallons of water. The flat rate model only worked because we generally could keep available capacity well ahead of demand. We could do that because we were adding customers like crazy, and customers sucking down high bandwidth were outliers. If those prerequisites ever changed we'd be screwed.

Well, here we are in 2014 and ISPs like Comcast are not adding customers like crazy anymore, because most of the people who want high-speed data from them already have it. But the customers they do have are, on a per capita basis, using much more bandwidth now than they were five years ago. And this is only going to get worse. The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever, but those requirements are no longer being driven by customer acquisition. Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.

But wait, doesn't Netflix pay for all their bandwidth already? Well, probably not: unless industry practices have changed drastically, even those backbone trunks are actually oversubscribed. If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. Netflix breaks that assumption.

I don't know quite what the solution here is, and I certainly don't want the kind of dystopian "the ISP only gives you access to web sites that you pay for" future the sharpest critics paint. But I don't see this as a step toward that dystopian future. What I see it as is the ISPs trying to figure out how to work usage into pricing models without saying "screw it" and actually adopting a usage-based pricing model.


You forgot to mention the part about how it actually does balance out because Comcast is receiving $18B billion a year for internet bandwidth from the its customers. Customers are requesting more data, but Comcast wants to be paid on both sides because they have a monopoly over the customer.

If it is really expensive to accept Netflix's offer of free hardware and free bandwidth then the additional cost should be passed to the customer. The problem is that it's not actually more expensive and Comcast knows their customers can't choose someone else so it's a much better strategy to squeeze the content providers.


Very, very well said.

Let's do some simple math to prove your point.

Assume Comcast has about 10 Tbps of traffic at peak, single-counting all traffic that transits AS 7922 (Comcast's national backbone network). [1] Now assume that a full 50% of that traffic is originated by Netflix, and the market price for transit is $2/mbps. The percentage is likely quite lower than that, and given 5 Tbps of traffic volume and coordination between the two networks to reduce the number of miles the bits need to be hauled by both networks, the per-megabit price is almost certainly quite a bit less.

But even with these obviously flawed guesses, we're only talking about $120mm/year in costs. Netflix's worldwide revenue in 2013 was $4.3b [2]; Comcast's 2013 revenue from their US HSI business alone was $10.3b. [3] If 50% of Comcast's $10.3b business was in jeopardy, don't you think they'd find a way to absorb that $120mm?

[1] http://as7922.peeringdb.com/

[2] Netflix Q4 2013 earnings release, http://ir.netflix.com/results.cfm

[3] Comcast 10-K filed 2/12/2014

(edited: formatting)


Comcast is prepared to provide that bandwidth to its customers subject to it's fair use cap of 300 Gigabytes. What they aren't prepared to do (for better or worse) is install larger circuits to Cogent so that Cogent's customers have unblocked access to Comcast's customers.

Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast. This will probably set a very bad precedent for them when it comes to renew those agreements in 5-10 years, or however long they will be. Maybe they are hoping for some competition to enter the US environment by then.


> Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast.

How would that work? Comcast customers by and large can't switch to another broadband provider and Netflix customers can certainly switch to another video provider (Comcast!). It's not like this is a brand new issue, Netflix has held out for a long time. The Comcast / TWC merger is what finally did it, 30M combined customers is too big to risk on a public game of net neutrality chicken.

Surely they're still working on lobbying efforts to make it illegal for a company like Comcast to simultaneously have a monopoly on selling broadband and compete with services using said broadband. It's pretty obvious at this point that internet access is a utility and it should be treated as such.


Completely agree with you here. It's time that the United States realizes that it's approach towards broadband was flawed and tries something new.


Almost nowhere does Comcast actually have a monopoly on broadband.


A duopoly or oligopoly isn't much better. There's still a marked lack of choice for the customer, which results in lock-in and higher prices.

The Hacker News readership includes many people capable of starting ISPs. Comcast's large profits, which have been used to fund a string of high cost acquisitions, are a juicy target for potential competitors. Yet you never see discussion of ISP startups on Hacker News because the barriers to entry protect the current carriers. This wasn't the case in the 90s, and had Hacker News existed back then ISP startups would have appeared frequently.

The issue of telecoms not having competition has been the case for years. Back in the BBS days Jack Rickard used to harp on this in Boardwatch [1] magazine. I recall him saying something like, "Two guys with a screwdriver should be able to start a phone company."

[1] http://en.wikipedia.org/wiki/Boardwatch


> This wasn't the case in the 90s, and had Hacker News existed back then ISP startups would have appeared frequently.

Indeed. The late 90s / early 2000s were the golden age of the CLEC. There were certainly pain points (like getting copper pairs provisioned), but I remember not caring that my building didn't support Comcast Internet because I had an SDSL connection from Telocity and an ADSL connection from Speakeasy.


A friend of mine had a pretty nice BBS setup in the early-mid 90's... It took him a dozen requests and 2.5 years to get extra phone lines in. That was definitely a pain point.


I actually knew a lot of small-mid sized ISPs that started off as BBSes... Have a T1 split for phone + data, with several phone lines, and a data pipe upstream. I used to subscribe to Boardwatch magazine as well.

That said, most were either bought out, or went under within a couple years. It was a rough time for commercial SysOps in the mid-90's... not that I really feel bad for them. Times and environments change... The BBS and art scenes are still around today, but not nearly what they were in the early-mid 90's...

I miss it a lot... since most BBSes were based in a local area, there were a lot of get togethers, and you'd meet and talk to people you wouldn't necessarily do in today's internet, which is based in interest groups.


This is true if you have some antiquated definition of broadband that includes a competitor offering "up to" 896kbps while Comcast offers everything from 1 to 20+ mbps, but false for everybody else.


Question: If Comcast is such a monopoly that everyone here keeps repeating ad nauseam, why hasn't it been broken up ATT style yet?


AT&T was broken up into a bunch of smaller regional monopolies that did not compete with one another. (Almost a century after its founding, by the way, and decades of on-and-off litigation.)

The cable industry has evolved into a bunch of regional monopolies that do not compete with one another.

The effect of breaking Comcast up AT&T-style would be... a bunch of regional monopolies that do not compete with one another. And then, as with AT&T, proceed to consolidate via M&A into larger regional monopolies... that do not compete with one another. Same as now.

Welcome to the American telecommunications industry.



That doesn't discount the fact that laying cable costs lots of money is a natural monopoly.


Wait, so in one post, you challenge HN's readership to prove that Comcast is a monopoly, and then two posts later you assert that cable is a natural monopoly. I'm confused what point you're trying to make.


Comcast is a natural monopoly, and there isn't anything wrong with that. Competitors are free right now to lay cabling to areas where Comcast already serves, they just choose not to because it isn't a smart financial decision.


http://nymag.com/daily/intelligencer/2014/02/why-comcasttime...

""" But there's a silver lining for people who hate this deal: By the government's own standards, the Comcast—Time Warner deal should be blocked. To understand why, you have to know that there's a thing in antitrust law called the Herfindahl-Hirschman Index, or HHI. In very basic terms, the HHI is a way to measure the concentration of a given market. And the HHI is the test the government's lawyers apply if they want to see whether a proposed merger would create monopoly-like conditions. """


What they aren't prepared to do (for better or worse) is install larger circuits to Cogent so that Cogent's customers have unblocked access to Comcast's customers.

That's totally backwards. Comcast's customers are paying for access to Cogent's customers, including Netflix.


That has never been how the internet works. You pay a part of the costs, the other party pay a part of the cost. When the other side opts for a cheap, poor quality connection (like Cogent), you suffer no matter how much you pay, and it's not your ISPs fault.

It's not Comcasts responsibility if Cogent did not have peering arrangement or had paid for sufficient capacity to handle their end of the connection, that their customer was paying them to provide.

If it was, I'd love to provide Netflix connection much cheaper than Cogent. I'll provide it over my 10Mbps uplink, and blame consumers ISPs when everything falls over.


It is Comcast's responsibility because I was paying them $1xx a month to take care of it. Bulk transit bandwidth is significantly cheaper than the price I was paying, so there's no justifying the claim that both sides have to pay.


Cool, let's be partners. I'll provide a service you want to access at 10Gbps, for an hour a week. I've got no connections to Comcast. I'll call up my brother, who runs a little ISP. I buy 10Gbps from him, and so have 5 other customers. You try to access my site. Packet loss ensues.

Since you've paid Comcast, they're now required to go to my brother's ISP and arrange more peering, right? Also, since you're paying, I shouldn't have to pay my brother, and he shouldn't have to pay anyone else, right?

Come to think of it, why isn't all hosting bandwidth free? Why do Azure and AWS charge for outbound bandwidth, but not inbound?


Let's bring this down to earth. You're confusing transit and peering connections.

Netflix pays level3 for transit service (you pay your brother). Comcast pays level3 (your brother) for transit service. Notice how level3 is neither the source nor the destination of the traffic.

Netflix's traffic increases from, for example, 500Gbps to 1Tbps. Comcast's level3 ports fill up because netflix increased in popularity.

In general networks that pass a lot of traffic like this end up peering directly and avoiding transit costs.

Comcast has two choices. They can either pay level3 for another 500Gbps or they can transition the full 1Tbps to free peering directly with netflix. And yes, I consider it Comcast's responsibility to do one of those two things with their subscriber revenue, which, incidentally, most other ISPs in the US are doing.

Instead they are choosing option 3 which is, leverage their local access monopoly to force netflix to pay them, and protect their cable and xfinity services.


Sure, if your brother's ISP has a 50Gbps pipe at a common peering location, and 10,000 Comcast customers all want 5Mbps video streams from his customers, Comcast should be peering with him.


"If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. "

This may be the cases for ISP-ISP connections, but every transit circuit I've ever purchased as an end-user has two elements, a Non Reccurring Charge (NRC) to set up the connection, and then a monthly recurring charge (MRC) consisting of, a port speed, and a 95th percentile metered usage charge.

e.g. I might get a gigabit port set up for an up front $5000 NRC, and then a 24 month contract for $2000/month port speed, and then pay $5/megabit @95th percentile of 5 minute averages sampled over a month.

The ISP samples the usage of my port every 5 minutes, (8640 samples in a 30 day month), drops the top 5% (432 samples), and then charges me the cost of the next sample (sample #8208) - If the sampled speed at #8208 was 1 Gigabit, and sample #8207 was 0 kilobits, then I am still charged $2000/month + 1000 * $5 = $7000/MRC for that month.

Cogent has always had a reputation in the industry for being the cut-rate player (to the point of other ISPs just dropping them as peering connections, resulting in companies literally not being able to communicate with their customers) - Netflix was trying to rely on Cogent to send data to their customers, presumably because they were charging significantly less than "Tier-1" transit-providers, and finally came to the realization that yes, you do get what you pay for.

None of this surprises me, and none of it appears relevant to the conversation regarding Net-Neutrality - this is a peering discussion, pure and simple.


That's one way to look at it. The reason there even is such a thing as a cut-rate service like Cogent is that ISPs like Comcast have a captive audience and can increase their transit fees with impunity. The CDNs that go along with that extortion are the ones that don't get labeled "cut-rate."


DS3 line? What is this, 1996?

Each individual Comcast customer has in theory paid for a DS3 worth of bandwidth (modulo the "cap"), which should be a reality-check that some of your assumptions from that era need updating.

The defense (and it was always a thin one) of ratio clauses in peering agreements was hot potato routing. The ingress traffic is what you actually have to haul a long distance, so the agreement says "you must meet us everywhere we peer, and you must maintain ratio." Together, those rules mean each individual path will be asymmetric but the overall load on the WAN will be shared between the peers. It was always slightly silly, though.

Hot potato is now over. CDN's are desperate to do all the work and bring traffic as close to eyeballs as the monopolists will allow because the cost of actually building a network is irrelevant compared to the rent the monopolists are extracting. The monopolists are selling so-called "transit" to haul packets 1 mile. They're really charging for access to their captive eyeballs, not for moving bits. If you want to send a few megabits to Asia, fine, whatever. You can do it for the same price. They're not even mad, "glad we could make a deal," etc. The situation is nakedly broken and warped, and everybody actually in the business knows it.

Peering has always been a mix of market power, shaming, and political excuses ("ratio" was one). What we're now seeing is the shaming and excuses fall away thanks to apologist bloggers and weak politicians, and market power emerge as the only thing that matters. Compared to this power, traditional outdated notions of neutrality are irrelevant: there is no need to give traffic less "preferential" treatment when you can simply depeer them and then try to sell them so-called "transit" to go 1 mile. "Paid peering" == "well, how much money you got? Let's make a special deal just for you".


I used DS3 as an example. But no, even in 2014, you've only paid for a DS3 (or whatever) worth of bandwidth in theory. It's not just the last mile that's oversubscribed -- so are all the trunks leading up to it. If that's changed between then and now, I would lay pretty good money that it's changed to make things worse, not better.

Without meaning to express either approval or cynicism, market power has always been the only thing that matters in this business. But CDNs have paid big ISPs special rates for years, because they really do put special strains on infrastructure. And my impression remains that that's what this particular battle is ultimately about.


Isn't flat rate pricing completely orthogonal to net neutrality? Net neutrality doesn't mean you can't charge me for usage, it just means you can't charge me differently based on the kind of usage, as opposed to the volume.

I personally think the resistance to usage-based pricing in the tech community is really weird, although I mostly see it on the consumer end. Hard caps are bad, and caps that only apply to some services are really bad, but charging more if you use more is completely reasonable.


The idea is that usage-based pricing doesn't make sense for customers because there is a hard limit on consumption simply based on the terrible state of last mile connections. Simply put, charging 2014 dollars for usage on a line with a capacity that was last relevant in 1985 is not motivated by actual capacity limitations, but a simple money grab.

(Not to mention that there are no inherent variable costs to transporting bits. Water needs to be purified, electricity is obviously usage priced - but you can be pushing bits forever once you have the hardware setup.)


I think the resistance to usage-based billing is practical rather than economic. With a flat-fee scheme you can look at your bill and say, "yup, $89.95 is what I agreed to pay each month."

With usage-based billing, the amount will float up and down and it's hard to understand how the amount is calculated. Usage-based telephone bills are broken down by calls, which you can probably remember.("What's this 46 minute call to 555-1234? Oh yeah, I called Mom last week.")

With IP traffic, how do you itemize the charges? Temporally? I used to get cell phone bills that included pages and pages of stuff like "Jan 27, 4pm, 239789987345 bytes". It was completely useless.

How about reverse DNS? That has other problems. • You use BitTorrent? Here's 247 hosts with incomprehensible names on your bill. • You use a VPN? A whole bunch of traffic is lumped together. • "Honey, what's all this traffic to nastyporn.com?" • It puts pressure on web sites to optimize their deployment strategy for billing clarity

The "correct" way is probably to log everything, then provide analytics software. But I can't imagine my Mom generating reports just to understand her internet bill.

Economically, sure, usage-based billing is a great idea. Alignment of incentives and all that. But practically, it's a pain for everybody involved.


I think the analogy to water or electricity billing is actually a very good one.

My electricity bill shows only X kWh per month. There is no "itemization" of usage. However I can purchase watt meters that can monitor power anywhere (including the main grid connection) if I decide the bill is too high.

I can definitely imagine a similar service sitting on the router or provided by the provider that does bandwidth accounting to deliver a picture of what is going on. I'm sure there would be many third party device manufacturers willing to do this traffic accounting as well.


If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7.

Whoa, whoa, whoa. This is exactly the assumption commercial customers paying for unmetered links are making, otherwise they wouldn't be paying for unmetered links. Not being oversubscribed is supposed to the entire point. If ISPs' business models depend on them being able to break their promises to customers paying for continuously saturated links that's their problem, but they're still on the hook for what they're being paid to provide.


I'm looking forward to chipotle_coyote response on this, but I don't think he's referring to the circuits that commercial customers pay for, but instead the ISP peering circuits - which are usually expected to run at much less utilization.


The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever [...] Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.

I dunno.. being profitable enough to buy time-warner cable for billions doesn't sound like a nightmare scenario from their POV. It sounds like they'd rather spend the money becoming a monopoly than improve their infrastructure.


You forgot to mention that network technology has leapfrogged last mile connections many many times over. In many ways, the problems have gotten considerably easier for many ISPs, and there are no capacity problems at big exchanges, and certainly no barriers to scaling up.


> Services like Netflix that send massive amounts of traffic in a single direction violate this assumption.

Netflix does not send any data. Comcast customers request lots of data from Netflix. Significant difference.

Another example (just in the other direction): A backup provider needs to pay up big-cable because their customers are uploading a lot of data?


Your second example is actually really fascinating - backup and photo providers get awesome deals on bandwidth because that part of the circuit is typically highly underutilized. So, ironically, it's a case where backup providers argue that their circuits should be charged at much less than market rates because the traffic is going in the opposite direction ,and their provider usually has circuits going idle in that direction.

I'm willing to wager that backup providers haven't run into any upload issues with Comcast for this reason.


you are some 5 yrs late... ISPs already put usage in the contracts.


The contention was that the change in traffic caused by the Netflix flows put Comcast & Cogent's peering out of balance.

Generally speaking settlement free peering requires that the arrangement be mutually beneficial. If the traffic exchange becomes inequitable then the side that disproportionately benefits would need to pay to cover it.

In this case Cogent was sending vastly more traffic towards Comcast than they were receiving (because of Netflix) so they were the misbehaving party.


Consumer ISPs always have received more traffic than they send, it's the nature of the service they provide. I'm not sure that the settlement free peering model used by upstream providers makes sense when applied to ISPs.

I'll admit my knowledge of peering is minimal, but consumer ISPs have always been something of an anomaly if memory serves. The traffic flow is highly one sided, but peering with consumer ISPs is obviously necessary if you want to actually reach consumers.


Yep, that's loosely accurate for consumer providers but Comcast does more traditional business traffic that one would think.

For example, they are the 30th largest provider globally going by the CADIA rankings (http://as-rank.caida.org/?mode0=as-info&mode1=as-table&as=79...) and have large traffic sources as customers. From the CADIA list this includes Peer1, Dreamhost, SoftLayer, Hivelocity, and Liquid Web.

As a eyeball ISP I occasionally get offered cheap transit from hosters looking to balance their traffic ratios. From the CADIA list it seems like Comcast may be doing the opposite.


I work in a totally different environment -- a very large enterprise IT environment with 200k+ users and probably 2,000 physical locations.

When we do business with cloud providers or other third parties, we usually start with internet based access. When the relationship gets bigger or needs to serve a large portion of our base, we typically either peer with that provider (in the case of big cloud providers like Amazon, Microsoft, etc) or establish dedicated connectivity between our networks.

I think Comcast, Verizon, TWC, etc has a point here -- as a relationship grows with another party, you should have a more robust connection. Netflix hasn't wanted to do that -- it prefers to use a shitty ISP (Cogent) for cheap, and augment it by co-locating content on end-user ISP networks for free. Services like Netflix and Youtube stretch the net-neutrality argument, because they aren't good citizens.

That said, the end-user ISPs cannot be allowed to discriminate, which their monopoly power will almost certainly enable them to do.


I agree. I am not arguing that Comcast should be allowed to go unregulated either. I strongly support net neutrality and do not want to see it thrown under the bus.

Assuming it wasn't harmed is also speculation and we should definitely be concerned about the deal and what it means for net neutrality.


Is it really peering in this case between Comcast and Netflix? How would Comcast leverage Netflix's infrastructure?


Comcast will likely get better customer retention, less service calls for higher quality netflix streams, a direct line to Netflix's core networking team to resolve problems, and a transfer of cash for the initial peering.

Comcast will also likely work with Netflix to estimate traffic growth and make sure their infrastructure scales before they have problems.

Again, we don't know the details of the deal. Assuming net neutrality was not damaged is also speculation. We need more information to know what is going on.


When Comcast has a packet that needs to get to Netflix, they can now hand it off directly to Netflix. That's all that peering means.


If this is a problem, then build physical infrastructure so your business isn't dependent on someone else's pipes. If states and municipalities put up roadblocks to it, then let's focus on tearing down those roadblocks.

I think it's ridiculous and self-serving for internet companies to complain about this. Getting bits to customers is a critical part of any online business. You can either do it yourself, or pay someone to do it. Does Amazon complain that UPS charges them to deliver packages to customers?


Both consumers and companies like Netflix already pay the infrastructure to transport those bits. The consumers pay their ISP to send and receive bits and the companies pay their ISP to send and receive bits.

The more apt analogy would be if UPS decided to make Amazon packages ship super slow unless Amazon payed extra fees to "upgrade" the shipping speed back to average.


That's not an apt analogy at all. Netflix connected to Cogent, which is one of the cheapest providers that still works (HE is cheaper but extremely congested). Netflix now determined that Cogent's connectivity to Comcast was insufficient, so they're connecting directly.

You could make the case that Comcast should have upgraded their peering with Cogent. That's the case in almost all of these disputes. But Netflix blinked first.

If you want to make useless shipping analogies, it's like Amazon ships using ABC to a rough wilderness. ABC uses simple cars that don't handle the terrain so they contract the local government to deliver packages in that area. You could say it's the local government's responsibility to fix the roads, or ABC's responsibility or anyone but Amazon's. Good luck.


Will amazons' customers be okay if UPS charged an additional fee to amazon's customers to receive their packages?


This situation is more akin to Amazon building it's warehouse on UPS's land to get shipments out (for a fee of course). The prior situation would be if amazon shipped via US post to UPS, then UPS ships to destination. Ideally the fee to use UPS (Comcast) land would be less than the fee to ship the product via US post (cogent) to UPS. The reduction in transit time also factors in to the decision Netflix made.


So if I need to drive to a customers site I should build my own interstate?


If you drive to your customers house over a private road, you can't complain if they charge toll.


Except your customers paid to have that road built to their house, why is there now a toll being set up (of which the customers get none of the money) to charge use of that road for something they want.


Internet companies already pay for their access: they pay their own ISP and hosting bills, just like Amazon pays UPS to ship packages.


They don't pay their ISPs to get their packets all the way to the customer. They're reliant on somebody else's property to get them there.


That is how the Internet works. You used multiple networks' property just to reply to my HN post. I doubt you paid all those networks, and I doubt that you could realistically use the Internet if you had to pay every network that stood between you and the websites you like to visit. Why should Internet companies have to pay your ISP when you do not have to pay theirs? Nowhere in the design of the Internet is your connection differentiated from an Internet company's connection.

Really, the point of the Internet is that you can communicate with other Internet users without having to worry about what ISP or network they are connected to or what arrangements govern that connection.


Internet companies that are also cable providers have an inherent conflict of interest; they want to keep as many people as possible tied to the dinosaur technology of cable as long as they can--so it's in their interest not to provide sufficiently good service to let people cut the cable, or to strong-arm networks into not providing their programming via the net.


That's already the case if you want websites like ESPN3 or NBC Sports Live Extra.


The difference is that it's the actual web sites making that decision. ISPs aren't blocking ESPN, but rather ESPN is making the deliberate choice to only allow access to their web site from certain ISPs. This is dumb, I think, and annoying, but there's nothing fundamentally wrong with it. Where you start having problems is when your ISP, who you are supposedly paying for access to the entire internet, starts blocking certain sites because you haven't paid extra.


Comcast wasn't blocking or rate limiting Netflix (in this scenario) - they had over utilized circuits with Cogent, who is a transit-provider that sells cut-rate transit of lower quality (and much lower cost) than other transit providers.



I got better and more reliable service from Cogent than I ever got from the likes of Level3, and having both been a Comcast customer AND seen the inside of Comcast, I don't know in what universe it could be called "quality".


I thought most did this due to licensing and 'blackouts' by the leagues. Although I'm not sure if this is correct for ESPN.


You don't have to pay extra for those services though. They come with your cable subscription.


My cable subscription is "Internet". I do not have a cable television subscription, and have no desire to have one; even back when Comcast was offering cable TV for $1 added to the Internet price and seemed shocked that I wouldn't want it...I replied, "I don't have a TV. WTF would I do with it?"


For a business...it was $300 cheaper to install Comcast with TV than without as of Oct 2013. I pay $2/month for TV but got free install and I can cancel TV in a year. It seems to me they need another mouth to feed.


Maybe I'm missing something but why would it be cheaper for Comcast to include TV?


It probably means that some mid-level manager at Comcast is catching heat from the home office for letting their TV numbers slip.

At least once a month for the past couple of years, Comcast sends me some junk mail begging for an "Account Review," in which they would add triple-play service to my business Internet account in exchange for a discount on all three. Those letters go in the trash on general principles.


You do pay extra. Your cable company passes along the cost of ESPN subscription to you, regardless of whether or not you want it.


You have to pay extra for a vestigial cable TV subscription because your broadband subscription isn't sufficient to access websites on the internet. Or you have to switch from a perfectly good ISP to an ISP that cooperates with the cable TV mafia.


That is not going to happen. The onus is not on the consumer, it's on the service provider. The service provider may transfer the costs onto the consumer but Comcast won't directly charge the consumers a premium. That would be a bad move and Comcast would lose out its customers to competitors.


What competitors? In many areas, they are the only broadband provider. The next-closest providers are typically DSL.

Consumers are going to get screwed on the front & back end of this.


Even after TWC and Comcast combine, they will be the only wired TV choice for only 9% of customers.


He said broadband, not TV.


9% of 300 million is a freaking huge number.


My guess is that in a true free market, a big chunk of those households would have no cable service at all. That is to say without the franchise agreements that gave them quasi monopolies in those areas, providing service wouldn't have been attractive in the first place.


Remember when there used to be five different cable companies listed in the weekly TV listings? Maybe you never saw that in your area, but when I was growing up in a semi-rural suburb, there were many cable TV competitors. Surely they didn't all require franchise agreements to want to enter the market.

Further, assuming franchise agreements exist and were necessary, consumers and municipalities should expect better from their franchisees than abuse of monopoly.


You're totally ignoring the difference between cable systems now and cable systems back then. Building modern HFC networks capable of supporting broadband requires tremendously more investment than cable TV systems of yore.


You said "wired TV choice".


> Consumers are going to get screwed on the front & back end of this.

Netflix is network heavy and competes with Comcast offerings using Comcast's own network. That was the key motivator here.

Even if Comcast has a monopoly in select areas, it cannot do price differentiation in just those select areas. IANAL, but that sounds illegal.


"Netflix is network heavy and competes with Comcast offerings using Comcast's own network"

Funny, back when I was a Comcast customer, I do not remember paying for "Comcast Network Access." I remember paying for "Internet Access;" sure, I went over Comcast's network, but I (maybe naively) thought that such distinctions were irrelevant on the Internet (that was kind of the point, if memory serves).


Its silly to pretend that the abstraction is the reality. There is no Internet, just a bunch of private networks connected together. No amount of wishful thinking will change that underlying reality. Now that real money is in play, the fiction of the internet is crumbling and the physical nature is becoming exposed.


"Its silly to pretend that the abstraction is the reality"

No more silly than pretending the the abstraction of "files" and "directories" is reality. No more silly than pretending that your computer is running a Python program, when in reality there are sequences of machine language instructions being run. Computers are useful because of abstractions.

"There is no Internet, just a bunch of private networks connected together"

You might have missed the definition of "an internet:"

https://en.wikipedia.org/wiki/Internetwork

There is, in fact, an Internet that we use every day. The overwhelming success of the Internet is due to the abstraction it presents, which is best-effort routing of packets from one host to another -- without regard for what networks the hosts are connected to, what arrangements govern their connections to those networks, and perhaps most important of all, what applications those hosts are running. On the scale of positive improvements to human communication, the Internet is up there with the printing press and written language.

"Now that real money is in play, the fiction of the internet is crumbling and the physical nature is becoming exposed."

No, what we are seeing is the impact of years of systematic attacks on the Internet by companies that are desperate to protect obsolete business models. Comcast is worried about the imminent demise of cable TV and all the power and profit that came with it. AT&T is worried about the demise of long distance fees, roaming charges, SMS charges, and all the power and profit that came with controlling the phone system. The recording industry and movie industry worked hard to leave the Internet divided into "consumers" and "services," even though the system itself was designed to be far more general.

In short, had we told these companies to adapt or die rather than tolerating their efforts to roll back the clock, we would not be in the situation we are in today. This has nothing to do with leaky abstractions, physical limitations, or the design of the Internet. It is just a power grab, a systematic effort to destroy a good thing that threatened the entrenched players.


> No more silly than pretending that your computer is running a Python program

Its a useful abstraction, but as physics causes CPUs to hit a performance wall on single threaded code, then you start banging up against the GIL and the failure of the abstraction to deal with parallelism. And when that happens you can't bury your head in the abstraction of a Python program. I'm not saying that abstractions aren't useful, but at the end of the day you have to concede that they are merely abstractions. When the abstraction becomes disconnected from the reality, reality wins.

> In short, had we told these companies to adapt or die rather than tolerating their efforts to roll back the clock, we would not be in the situation we are in today.

Adapt to what? Abstractions must adapt to reality. Reality does not have to adapt to abstractions. You can of course invoke the government, to regulate the reality into conforming to the abstraction, to the benefit of certain companies and the detriment of others. But history has not been kind to regulatory regimes of that nature. The deep irony of your position is that its the same thinking that led to the current mess with cable. In the 1980s, cable was the future. People invoked the potential to distribute educational material and public access material to the masses, and decided that such an important technology should be regulated. So they created the cable monopolies, under the conditions that they have cheap basic access rates ($13/month in Phila), carry public access or educational programming, and build out even to poor and rural communities. And it was an utter disaster in retrospect.


run a search at broadbandnow.com, see what you get


Yeah, my building here in NYC, I can get Time Warner or DSL (Verizon or from a reseller). That's it. Lots of us are stuck with 1 or 2 crappy options. I have overpriced Time Warner and a Verizon hotspot as a backup because the TW cable connection goes down about once a month. The other option is DSL which is too slow to do work and was down a week of my 30 day trial when I checked it out years back.

Keep in mind that this is New York City. You'll have lesser options in many other places in the US.


I'm in Austin, TX.

We have two FTTH providers (AT&T, Grande). Grande offers 1Gbps now, AT&T is 300Mbps now, 1Gbps this summer. Time Warner offers 300Mbps now (DOCSIS).

Google fiber is being deployed this year (1Gbps).


Yeah, funny world - once one decent internet provider pops up in an area (Grande, presumably, and now the thread of Google Fiber too), all the other classic providers seem to provide the best service they offer throughout the country. AT&T doesn't offer that in 95% of their territory in the US (so like 99% of the US can't get it), and the same goes for Time Warner. If any area has decent internet options, it's not because of a classic / well known broadband ISP, it's only because of some small independent ISP that forced a local market to be competitive.


Austin is the best case scenario in the country.


The terrible consequences of competition.


broadbandnow.com isn't fine-grained enough to be trusted.

For my zip code they list AT&T UVerse, AT&T DSL, Cox and Time Warner. The reality of my actual address is that my only options are Time Warner and AT&T DSL (old school fully copper DSL, not UVerse). I'm positive these are my only wired options because I have researched this extensively (including actually calling the companies, not just relying on bad Internet data) due to being unhappy with my two options, and I continue to look into it about once every 6 months but nothing ever changes. Both AT&T UVerse and Verizon FiOS seem to have gone into deep slumber mode when it comes to service expansion.

The really sad part is that I've been using AT&T DSL for the past two years because while it is significantly slower than Time Warner cable in the bandwidth department, it has been far more reliable. Cost-wise both are a complete rip-off already, and if the government doesn't step in on the ISPs soon, I expect that the cost issue is going to get a lot worse, not better.


Yes, this site says I have Cox and Comcast in my zip code. I can assure you that Comcast is not doing cable internet business in my zip code or anywhere near my zip code. I don't believe they have a presence anywhere in the county.

Also the information blurb at the bottom says I have a fiber provider, but none are listed.

Finally, it's missing some local providers which I am positive exist; our local power/water provider also provides Ethernet to local business and some residential areas, but it's not listed.


I did this and keep doing it periodically. I only have 2 (but and I feel lucky). I keep bouncing between them to get "new customer" promotions for 2 years or so. Then I quit and switch to the other one. Sometimes talking to the "retention" department results in some minor bill reduction. But no, there are no 10 companies to choose from and picking the best one.


The reason Comcast would rather have Netflix pay than pass those costs to consumers is because they don't want to make net neutrality any more salient for consumers. If Netflix pays Comcast then it goes mostly unnoticed by the average person. If people have to start paying then the people screaming for net neutrality will start to scream louder, and Comcast really does not want that. It's not about competition with other ISPs, because as other posters have pointed out, Comcast is usually the only ISP in many markets.

Comcast worries about the FCC and Congress, not its competitors.


Comcast worries about the FCC and Congress, not its competitors.

It would seem that worry extends only so far. In this case, if what you're saying is accurate, Comcast intends to take (i.e., extract rent), and it just doesn't want to get noticed while it takes. Fear of the FCC and potential consumer backlash do not extend so far as to help it to reconsider its basic strategy of expanding rents, even as its merger with Time Warner is under review. There's a certain boldness to Comcast's move in working out this deal with Netflix.


Pass on the costs? Isn't this supposed to lower costs for Comcast even without getting paid by Netflix? If so you mean pass on the more profit they want.


You're right. There are no actual costs, and shame on me for adopting Comcast's language.

ISPs have the option of making content providers(Netflix), or content consumers(you and me) pay more. They're choosing content providers because they don't want to make net neutrality more salient for their customers.


Will that really hold up, though?

Netflix isn't just going to absorb those payments, they'll have to pass them along to customers. Customers will want some explanation for why their bill is going up . And if Netflix's per-user costs become substantially different at different ISPs, they'll have to start thinking about ISP-specific surcharges...


The problem is that in many Comcast markets there are no viable competitors.

With Comcast I get ~3MB/s internet. DSL and Wireless broadband aren't even close to those speeds (in my area). So if I'm forced to pay a premium or switch, I have to "give up" significant performance.

PLEASE GOD GIVE ME GOOGLE FIBER.


Not when their competitors will do the same. Heck, what competitors? Many areas only have Comcast service!


>> "Many areas only have Comcast service!"

Do a lot of people in the US only have access to one or two ISP's?? If so why? I'm in a not particularly urban part of Ireland and I can choose between at least 5 ISP's, decent speeds, and a good variety of prices.


The answer is, it's complex. It really depends heavily on where you are - and streets matter. But many people only have one or maybe two choices for wired, broadband Internet: DSL or Cable.

I live in a suburb of a medium sized city. At my house, I have 3 choices for wired Internet access: Knology/WOW, AT&T DSL, AT&T Uverse. DSL is right out because it's too slow, so really it's a choice between Cable (Knology) and AT&T's Uverse (which is FTTN or FTTP depending on various factors).

We moved here a couple years ago, two miles away from our old house. Same city, same suburb, just 2 miles difference. At our old house, we didn't have Uverse as an option - it was just slow DSL through the phone company or a single cable provider. If I had lived three streets over (less than 100 yards), I would have had a choice between Comcast or Knology cable, and DSL.

In other areas of the suburb that are further out, you start to see Mediacom (another cable provider). There are probably some areas that Comcast and Mediacom overlap in.

I have a coworker that lives in a rural area, and all he has is Windstream DSL. Cable doesn't reach out that far.


http://www.broadbandmap.gov/number-of-providers/wireline/ser...

Unfortunately, the min/max slider forces hard-coded values during page load, so I can't link to specific graphs. I suggest checking out "served areas"/min:1/max:1 and "unserved areas"/min:2/max:whatever.

Those maps suggest the "1 ISP" set (or any set, really) is fairly uniformly distributed across the nation. In my opinion, it's not quite as biased towards the urban/rural divide I expected.

http://www.broadbandmap.gov/summarize/nationwide

    #ISP %USA
    0    3.3%
    1    8.9%
    2   32.0%
    3   35.6%
    4   14.3%
    5    4.1%
    6    1.3%
    7    0.4%
    8+   0.2%
So it's a minority, but a substantial one. I would also guess that at least "some" of the 2-ISP group have various barriers that make it de facto 1-ISP.

Regardless, 8.9% of the nation is still a LOT of people, and "2 or 3 providers across the nation" isn't really a thriving economy full of healthy competition.

The interesting data points, I think, are the high end. A 4-ISPs isn't THAT different from 1-ISP (8.9% vs 14.3%), and those numbers drop off fast in the 5+ range.

Oh, and yes, I'm excluding radio, in any form ("G[0-9]", various satellite/asymmetric services, etc). Those are a different product entirely, and any similarity in feature set is only superficial. You can't play games or any other low-latency task over most wireless, and VoIP doesn't like the "several second" latencies of satellite, for example.


The last mile. (http://en.wikipedia.org/wiki/Last_mile)

Most places are wired with one phone line, and one cable line. That phone line is generally owned by one specific phone company, and that cable line is generally owned by one specific cable provider.

So those are the choices of ISP: you can choose cable service or dsl.

For a few years the last mile over phone had to be shared with other IP providers, so you could choose between a) one cable company, or b) a few IP over DSL ISPs, but that ended sometime back.

There is also satellite providers, but they usually, I believe make a deal with your phone company, wireless, which is expensive, and fiber, which to my understanding is still just an upgrade from the over cable company.

How is the last mile provided in Ireland? And what are your choices of ISPs?


It is common in Europe that owners of the phone lines must give other ISPs access to it (for a - regulated - price). This in turn stems from the fact that the owners of the phone lines are (or used to be) owned by the state.


>> "Most places are wired with one phone line, and one cable line. That phone line is generally owned by one specific phone company, and that cable line is generally owned by one specific cable provider."

That seems to be where things differ. My phone line is owned by BT but I can still order internet through a different company. I pay that company an extra £10-15 per month for line rental (so I guess BT, even though they own the line chooses/or has to rent it to others).


Yeah, we had that for a few years during the 90s. I forget when we lost that, but it wasn't good.


[deleted]


I can't recall the last time I even heard of somebody in the US having the choice of more than two high-speed ISPs. At best, your choice is cable from one company and DSL or fiber from another. I'm sure there must be some places where there's three or more, but they must be extremely rare.

There are more if you count wireless, but those don't really work as a substitute for real home broadband, even with companies (e.g. Clear) that pretend they do.


Ireland and Britain enjoy better competition since they're a smaller area. Australia and the US OTOH are terrible since the population is so spread out (Australia's slightly better as the population is mainly coastal).


That's a cop-out. Services in the UK are either over the Virgin cable (HFC) network, or BT-owned copper/FTTC. The reason there's competition is a solid system of regulation which ensures fair and reasonable access to the infrastructure.


That's quite a different posture though. BT was a government owned corp, and when it was privatized, investors bought it subject to the restrictions in its license. In comparison, the cable companies built their infrastructure against the backdrop of deregulation, in which it was illegal for them to be granted exclusive franchises.


If this defense had any merit, then there should be far better options in large American cities. This is not the case.


No Britain enjoys better competition because BT's final mile is heavily regulated so that competitors can offer services and are charged regulated prices for access to the unbundled local loop.

The BT Group was also forcibly split into separate operating companies, particularly OpenReach which owns the local loops and sells services to all ISPs including BT Retail.


Most US municipalities have a franchise agreement for a particular service. The franchisee has exclusive rights to provide a service in the franchised area. So there is often one cable tv provider and one wired telephone provider in an area. This naturally leads to one cable internet and one DSL provider. Some areas do also have a fiber provider (Verizon FiOS, Google Fiber).

These same areas are often also serviced by wireless providers, either cellular, satellite or some other kind of fixed wireless.

Note that while DSL tends to be available in every market, it may not actually be available at your home due to distance from the CO. Or it may be available but you're so far away that the performance would not be worth what it cost.

So the problem is that there's a clear hierarchy of what's good in any area. Where I live it's cable > DSL >> wireless/satellite. Compound this with the pricing structure: where I live it's wireless/satellite >> cable == dsl, with cable having some speed tiers that DSL can't match. At the end of the day, for any consumer trying to make a decision on value and performance, there's often really only one option.

I'm going to assert that this is how it looks for the majority of the geographical US. I'm not sure about the larger metro areas.


I have a choice between Time Warner's Road Runner cable based broadband, or a fairly lousy DSL provider.

I live in a middle/upper-middle class suburban area in upstate New York.


Are those all some form of DSL? Is there a big choice of phone companies? I know in the UK, its BT and a couple of resellers. Do you still have to pay for a phone line rental to someone else?

In the US, you generally have a choice between one (two if you're lucky) cable companies and DSL.


You have to remember that, compared to the US, Ireland is tiny. The majority of the US population does not live in large urban areas, who while New York for example might have lots of choices, Spokane probably doesn't.


While this is true, it doesn't really explain the situation in the US, IMO, since the very populous urban centers here suffer as badly, if not more so, than the more rural areas.

If you live in a sparsely populated area the excuse is "well, it is expensive to wire you up because of distances". If you live in a densely populated area the excuse is "well, it costs a lot of money to put new infrastructure into a densely populated area".

In both cases, lots of other countries are managing to do both of those at far lower cost to the consumer than occurs here in the US. Ultimately the ISPs here in the US are just greedy, duopolistic fucks.


New York does NOT have choice. I live in Manhattan (Upper East Side) and have the choice between $60 1/15Mbps Time Warner Cable and $40 1Mbps Verizon DSL. Theoretically some people in the city have access to Verizon FiOS but I haven't met these people and complaining about ISPs is pretty common for the locals.

The only factor that determines the quality of your ISP options is having a competitive player in the market.


Depends.

Many people in the US have quite a bit of choices: Cable, DSL, Cell+Internet, Wireless. Usually those that live in large urban areas. But the service and quality of these ISP's varies greatly, which Comcast--in my area, being top.

Those Americans that live in rural areas, may not even have access to high speed internet, or always on internet--see the Xbox 1 fiasco for more details.


"Many people in the US have quite a bit of choices: Cable, DSL, Cell+Internet, Wireless. Usually those that live in large urban areas. But the service and quality of these ISP's varies greatly, which Comcast--in my area, being top."

This needs to be emphasized more. We rarely get to choose between equally cost-effective ISP options. The typical choice is one cable company, one phone company, and some kind of wireless service (possibly satellite). That is not competition, any more than heavy truck sales compete with mo-ped sales.


Same here. I'm in Bangalore, India, and I am spoilt for choice! There are about half a dozen ISPs that are recognized nationwide and then there are about a dozen that are specific to Bangalore with comparable QoS. And I also hear that ISP costs in US is really high in comparison. I really wonder why do they not come under anti-trust laws.


There are normally only one or two low cost options. There are often some higher cost options. For example, in the Bay Area there is high speed microwave internet available, but it comes out to more like 150/mo.


The costs always inevitably transfer to the consumer though. It may be opaque how things are truly priced but this is the way it works.


Why not? Isn't this exactly how they sell premium channels, like HBO?


Not really the same.

HBO charges Comcast a flat rate to carry their content, because HBO has great content.

Comcast then bundles HBO in various packages designed to bring a profit, knowing that people just want HBO, and are willing to pay extra for a bundle in order to include it.

In this case, Netflix is paying Comcast for the rights to use their End of Mile network.


In most parts of the country is Comcast had viable competitors , Netflix wouldn't have to pay it. The problems is cable and broadband industry is US is controlled by very few players. In quite a few areas, there is simply no choice (Satellite is just too expensive for many people),


This was from 2009; it's more prescient than ever, and no less terrifying: http://tctechcrunch2011.files.wordpress.com/2009/10/net-neut...


> I don't see any other way around this.

Vote with your feet.


So I should just disconnect myself from the internet?

Because it's either Comcast or a DSL provider that won't give me a straight answer as to whether or not they service my address. And this is in Boulder.


Interestingly, Boulder forced a few years ago the electric utility (Xcel) to bring fiber to the vast majority of houses in town, but the fiber sits unused because it was intended for realtime metering solutions and the town & utility gave up on that goal.

Imagine if that fiber were used as a community network! Unfortunately, the town is not planning to do this -- they've excluded the fiber from their plans to take over the Xcel infrastructure.


It's not really a free market to end consumers. To change cable companies, I have to go for some WiMax/Wifi type ISP (lower quality in most cases), DSL (also lower quality in most cases), or move my residence to somewhere in the service area for another cable company.


That's not an option for entirely too many people to be meaningful.


Exactly my thoughts.


You don't see any way around this?

I guess you've never heard of VPNs, then?


Wow, Netflix has decided to negotiate with the terrorists? I'm pretty disappointed.

Edit: Having read more of the non-paywalled version, I have to say I think they're shooting themselves in the foot, even if Comcast was offering a better price than Cogent. It sets a terrible precedent, and gives Comcast the leverage to bump the price extortionately at the end of the deal.


You don't understand.

This is the creation of an economic moat (a term coined by Warren Buffet.)

The gatekeeper creates a toll and the tradesman agrees to pay the toll. This is an entry cost to new market players and therefore reduces competition. This toll may be increased as the biggest player (Netflix) grows in size and this benefits them, as eventually new startups have $1+ million startup costs and nobody is willing to invest in that cost-payoff scenario.

This is a win-win negotiation for both Comcast and Netflix but a loss for consumers. Unfortunately it's just how the game is played.


This is an interesting perspective but I think a bit too startup-centric. Startup costs for a Netflix-like company are already in the tens of millions to negotiate license deals. Netflix is probably far more concerned with Amazon (and maybe Google) taking their business away.


True.

But I framed it too specifically. My perspective holds true at the level of large organisations wanting to take punts towards similarly large markets. Signing the money off is going to become more and more difficult when the access cost is so high, and barely anybody will want to take the reputational cost of failure to enter the market.


This is more like a reverse moat: you only have to pay it after you succeed. New entrants don't have to pay the tax -- peering without money changing hands will happen to "just work" for them just like it used to. Only after they get big will Comcast create a new traffic jam and send them a quote to fix it -- that's too late to help Netflix.


Good refutation. You are correct if they do not extend the tax to new entrants (which I had forgotten was currently the case.)

Still, perhaps Comcast will deem all of the companies big enough to compete with Netflix large enough to pay the tax.


+1 Remember when online video was RealPlayer's website? Imagine if RealNetworks had paid Comcast, and a startup called Netflix wanted to show their product to Comcast's customers. There wouldn't be a Netflix.

Whoever has the market right now has it extended for a long time.


It's not a win-win, since Comcast itself is a competitor to Netflix. You only have one side paying the tax.


They reduce the cost of having to fight new market players, which is actually far more difficult to control. It's now just a question of what leverage they will have to play against Comcast which is potentially a much easier game.

Generally the bigger you are, the more you will want to avoid volatility as big tends towards fragile - a big company will prefer static contractual obligations to another big company over being attacked from all sides.

(I am unsure however how this game plays out in the later stages if they lose leverage and entry costs increase to the point that it begins to suffocate the market.)


Once upon a time, if you wanted to trade in your borough, you'd set up a stall and start trading. Your startup costs would be the materials for your stall, and your goods to sell. Eventually, people started "owning land", and in order to trade, you'd have to pay your local baron a levy, tax, or rent. This status quo has been maintained since, and mobility within the structure only arose as a result of vast technical and ensuing societal change (the industrial revolution).

We are watching the ascension of the new barons. They will hold their privilege with an iron fist, for as long as they hold it, they define the playing field. They would be foolhardy not to.

The barrier to entry will indeed rise, as peering costs will end up passed on to content producers rather than consumers - consumers will not accept the cost rises, and market competition on the ISP side will ensure this doesn't arise - but on the supply side, large players can and will pay for an advantage, and small players will either be subsumed or destroyed. New entrants will need to be well armed indeed.

Now, this isn't quite the same as the feudal picture as previously described, as the barriers will at first be monetary rather than cultural, and have not yet had time to thoroughly entrench. If this is allowed to persist as is, however, without resistance or concern, we will indeed find ourselves in a entrenched technocracy of the foulest dimensions, with very little recourse for anyone without access to vast capital.


I'd have to string strongly with this.

Everything we know about Netflix is that they're a data driven company. Chances are it's far more likely they have seen viewing rates drop for Comcast customers who were affected by the slow-down.

If a typical end-user sees their internet working normally, bu see slow-downs selectively to streaming to sites like Netflix, chances are they're going to wrongly attribute the slow-down to the service and not their internet connection.


It's very easy for netflix to educate the users about whose responsible for the issue.


Ok so they are educated, now what can they going to do about? Nothing.


If that's the case we really should reconsider our support for netflix. Cancel your subscription.


> negotiate with the terrorists?

The term you should be looking for is “paying the danegeld”:

That if once you have paid him the Danegeld, you never get rid of the Dane.

— Dane Geld, Rudyard Kipling


That sounds like an old-fashioned term for this:

http://en.wikipedia.org/wiki/Protection_racket


Thank you for introducing me to that fantastic poem. In case others are interested as well: http://www.poetryloverspage.com/poets/kipling/dane_geld.html


I actually linked to that in a comment just before you posted this. Synchronicity :)


As much as I hate the ISPs, calling them terrorists is a bit too much.


I'm just using a metaphor. It's all business, but the game theory version is that by even engaging with them here, you legitimize the tactics and have to continue to pay up every time anyone comes calling.

Kipling has a cogent explanation of it in poetry form:

http://www.poetryloverspage.com/poets/kipling/dane_geld.html


Actually, the way I see it, further inflation in the word 'terrorist' can only be good.


That sounds like something a word-terrorist would say.


It is too much to compare them to what terrorists do, but let's face it - this could lead to more damage to the internet than terrorists could ever do.


World economy as whole. This kinds of stuff just strangle innovation. Currently I am helping (remotely) a friend in US to move his startup (creating value in US) up to speed while bringing hard cash in my country. Both the startup and our workflow rely on the openness of the internet.


I don't know. The word terrorism already doesn't mean anything anymore thanks the government and media so I think it fits just fine.


Really? There are web pages of terrorists that use the ISP infrastructure to send and receive data. Guilt by associations was 3 or 4 hops according to Snowden files? Since the ISPs refuse to be common carriers it is their fault. (Mostly sarcasm)


Masses of humans, whether it be the Internet or IRL, have a disgusting fetish for hyperbole.


Easy there HITLER

;]


Terrorist use terror to further their (often nebulous) ends. Comcast just terrorized me into seeing beloved intarwebs reduced to the absolute shithole that cable television has become. That's terrorism to me.

I suggest we treat them accordingly.


It could simply be a way that they're limiting losses - and will sue them later to change it, get money back, etc..


This might not be an anti-net neutrality deal. Here are the details we know at this point:

1. Netflix is currently using Cogent

2. Comcast offered them a direct connection to their network via paid-peering.

3. Netflix agreed and will move over.

4. Nothing about net neutrality was mentioned in the article.

This deal is being referred to as paid-peering. Peering is a practice performed at ISPs where two ISPs hook up infrastructure and agree to maintain their sides without charging for bandwidth. Peering is a common practice among the industry, and content-providers are always looking for ways to peer with more ISPs because it drives down their overall cost of operations.

The article says nothing about net neutrality. We don't have enough information to say whether net neutrality was harmed in this deal. We can only speculate at this time. Hopefully we will hear more from Netflix on this.

[edit to fix list formatting]


You don't seem to understand: This is not a technical issue; is a conceptual one, people are not going to read the details of the deal, rather to grab the idea and slowly believe that paying for better connection with a site/ISP is normal.


>rather to grab the idea and slowly believe that paying for better connection with a site/ISP is normal.

Is is normal. ISPs are not equal. There's a reason you pay Level(3) a lot more than HE.


What's the real difference between holding peering connections for ransom versus implementing packet-level QoS?


This is actually good for Netflix and terrible for any competition. While Netflix probably has to hand over a bit of money, they are in a much better position to do so than any of their competitors as they're the biggest. In effect this creates a massive entry barrier into the video streaming market. You will now likely need to set up a massive infrastructure and pay massive bribes to various internet providers to be able to compete. So this is probably a smart move by Netflix by creating entry barriers that they should be able to easily pass on to consumers if they can achieve market dominance.


This sounds like standard paid peering; there's nothing non-competitive here.

If you're a small provider, you tend to buy transit from one or a few providers, and pay per bit in whichever direction has the most traffic.

If you're bigger, maybe you can setup peering. If you have relatively balanced flows, and meet other qualifications, you can usually get free (settlement free) peering. If your flows aren't balanced, then the tradition is that whoever is sending more pays. If the networks you're establishing paid peering with are cheaper than your transit provider (usually the case), then peering is still a win for you, and a win for the other network.

If Netflix wants to get settlement free peering with Comcast, they need to balance their traffic to Comcast (maybe partner with a backup service, or web crawler)


The tradition when backbones peer with each other was for traffic to be balanced. When a content provider peers with an eyeball ISP, free peering makes sense regardless of balance.


That makes a lot of sense. Not solving this earlier is a detriment to the customers of both companies.


The same thought came to my mind, a moat around Netflix's current business. However, I don't think that's quite how it will play out.

Comcast won't charge new streaming providers from the get go. That will stunt its own business opportunities. Instead it will prefer to let them grow to the point where they can claim network congestion issues. At that point it has a "justifiable claim" as well as more negotiating leverage.


Yeah, but nobody is going to go into the streaming biz because of this. Netflix is scaring off would be competitors.


If you want to get into the streaming content I think a tax on distribution will be much lower down the list of concerns. Content licensing and infrastructure setup is the bigger challenge. The players in this sector all quite major (Apple, Amazon, Google).


Or you could just use an existing CDN that's already paying the bribes.


Bingo. The real story here is that Netflix is trying hard to get off third-party CDNs and that's what is inflicting a lot of the problems lately. I'm sure they figure it out eventually, as they found one solution here, but the internetworking business is a lot harder than it seems on first glance and not something you should try to wing.


Not so much established and highly capitalized competition (like Amazon), but disastrous for startups and new entrants. It adds a major barrier to entry in the form of a b2b paywall before you can reach consumers.


Isn't that's already the case by default? My read of this is that they're paying Comcast directly because it's cheaper than the conventional route of paying for commodity transit via a regular ISP (though more expensive than their preferred solution, a no-fee peering arrangement). As a new entrant I don't have much choice but to pay for commodity transit: Comcast is not going to peer directly with me anyway (or let me park my CDN boxes on their network, for that matter), so I need to pay the transit ISP. If I do so, am I at any disadvantage? Afaict, if I'm a new entrant who sets up colocation at he.net, say, and pay he.net for transit, I won't have any trouble getting my data to Comcast users.


Once you pay the danegeld you never get rid of the dane.


Any competition other than Comcast.


Not entirely sure why this is such a big deal. Peering agreements happen all the time. Lots of large services peer directly with consumer ISPs because it's cheaper and more efficient than using intermediary transit.

This has nothing to do with net neutrality. Comcast is not throttling or discriminating traffic in any way (at least not publicly). This is a peering dispute settlement.

Does this affect competitors to Netflix? Depends on how much volume they have. If their volumes are low, they can still reliably use transit providers more economically. As their revenue increases, they'll need their own backbone network, but their ability to pay for it goes up too.

Is Comcast big, bad and evil? Probably. But not in this case. Comcast dealing directly with Netflix is good for both parties. They are entitled to charge Netflix to peer directly because most traffic is going to flow into Comcast's network from Netflix. They also probably had a case negotiating away from settlement-free agreements with Cogent for the same reason. Again, none of this is out of the ordinary, ASes negotiate peering deals with each other all the time based on the volume and direction of the traffic they exchange with each other, and this is all fairly unregulated.

Net neutrality is irrelevant in all of this, it only matters if Comcast actively discriminates between traffic at their end. This deal is not connected to the ruling.


Comcast is a residential ISP. What possible traffic could there be to flow out of their network, other than requests for other data? By that logic, everyone has to pay to peer with Comcast because "Ha ha, we have all the customers. Darn, it sure is a shame that our customers don't have any data to send to you. Darn it, we really wish we could do free peering, but we just can't, aw, darn it. [cue South Park cable company responses]"

No, what happened was Comcast discovered another place they could fuck over the customer - by charging for peering even though it benefits their customers to not do so, and decided to spin it into some way that makes people feel like Comcast was somehow being injured by Netflix under the old settlement-free peering model.


While Comcast is primarily a residential ISP they do have a non-inconsequential number of transit & business customers that have more balanced or outbound heavy traffic flows. It most certainly isn't enough to balance out the residential traffic but they are trying to grow that end of the business.


Neither Comcast nor Netflix are trying to operate a non-profit, it's a payment dispute between two large companies who disagree over who should pay. You could just as well argue that Netflix was letting Cogent / Comcast fight it out instead of directly peering with ISPs which benefits customers.


The Biggest problem here is, I as a Comcast customer have already paid Comcast to bring me the packets from Netflix... That is what my outrageous bill every month is for..

They will now be DOUBLE BILLING for the same packet, a cost that WILL BE passed along to the netflix customer.


But what about the Comcast customers who don't use Netflix? They win from this deal! At least, if you make the assumption that costs are passed on to the consumer, as you seem to be making.


Costs will be passed along on the netflix side

Savings(if any) will not be passed along on the comcast side

Personally I do not believe there will be any real saving for comcast in the first place, so there is nothing to pass along, it is 100% profit for them


Previously Comcast argued that, I believe it was Level 3, was "pushing 5 times more data onto their network than Comcast was pushing onto theirs"

This made me laugh seeing as every one of Comcast's customer ISP packages have 5 times higher download speeds than upload speeds.

It's almost as if Comcast is forcibly causing an imbalance of traffic so they can extort more money out of Tier 1 providers.


Beyond looking at the implications for Net Neutrality (since we don't really know the implications of this deal going forward in relation to this); for people who are wondering why this has happened, this looks like this is part of Netflix's Open Connect Project[1] which is a single-purpose CDN network - and is essentially a paid-peering deal.

As part of the Open Connect Project, Netflix has deployed servers which are hundreds of terabytes in size in strategic locations all over the work and they contain ~60-90% of the data required for that particular country.

These devices either live in the ISP (which is what Netflix wanted to happen with Comcast) or live in an internet exchange where multiple ISPs can connect to the Open Connect devices.

There are 3 core benefits in doing this:

- Better Streaming due to fewer hops to the client

- Netflix avoids rate limiting (if they're inside the ISP Network)

- Peering is free (usually) and is essentially where two ISPs connect their infastructure and agree to maintain their own sites without bandwidth charges. It's common throughout the industry and content-providers are always looking to peer with ISPs to drive down their overall costs of operations.

[1] http://openconnect.netflix.com


Netflix does not use AWS to move video bits, but yes it will save them money on CDN costs.


Are the insane? Why would they agree to this weeks after the net neutrality case, and not wait at least until they see what Congress and FCC want to do about it?

The only other explanation is that they may actually like this, because they know it would slow down a lot of their smaller competitors coming after them. It's even stranger when you think that customers must've blamed Comcast for their poor Netflix experience lately, not Netflix.


How would Net Neutrality help here? Comcast wasn't throttling Netflix vs. other content, they were refusing to build bigger pipes to Cogent.


It looks like Comcast was refusing to upgrade specifically to hurt Netflix, which falls under the sprit of net neutrality if not the letter.


Net Neutrality requires that you treat all traffic the same, regardless of source/destination - it certainly doesn't require that you create larger connections without being paid for them.

And, let's face it - anybody who has ever used Cogent for transit (usually at 1/3 to 1/10th the cost of other providers) knew what they were getting into.


I don't think Cogent's cheapness has much to do with it. When Netflix was using Tata, Comcast refused to upgrade their Tata links. When Netflix used Level 3, Comcast found a way to create congestion there.


Equivalently, they were refusing to upgrade specifically to help Netflix.


Comcast customers are paying for uncongested paths to the whole Internet, not just 70% of it.


Does that include the Galapagos islands or a server in someone's garage that gets slashdotted?

You can't really guarantee uncongested paths at all times to the "whole Internet".


> The only other explanation is that they may actually like this, because they know it would slow down a lot of their smaller competitors coming after them.

There ya go.


Netflix could also be seeing declining engagement or subscription metrics for customers on Comcast.


I already had thoughts of dropping netflix due to their streaming quality. It's gotten so bad in the last three or four months that it was like I regressed to watching downloaded xvid's.


Because their business cannot wait...

It's like a hostage situation; you say "yeah I don't negotiate with terrorists" but when you see your wife/kid/so threatened and the money is just there in the bank it's hard to hold on to that posture...


Netflix and other streaming services should not engage in these types of contracts. They should instead detect slowdowns and notify customers of the slowdown and to call their provider to fix it, resulting in increased support costs, customer attrition, etc for the provider. Peering should remain a 50/50 split between the major peers. Just because Comcast and Verizon are sitting there refusing to turn the ports on until you cough up $ (basically a hostage negotiation) doesn't mean you should.


> Just because Comcast and Verizon are sitting there refusing to turn the ports on until you cough up $ (basically a hostage negotiation) doesn't mean you should.

Would you please enlighten me as to why an ISP should be required to provide access to a resource they aren't being compensated for?


Because their customers are paying for it.


Nope. ISP customers pay for last-mile infrastructure and access to the connection that the ISP has brokered with larger traffic carriers, whatever that may be. You should read up on peering agreements.


And part of that payment is for the ISP's peering agreements with other providers. Why would I pay just to connect to Verizon? I want my connection beyond Verizon to the peer networks hosting Netflix, Facebook, YouTube, etc. No customer would only pay for the last mile if that's all they got.


Yes, part of that payment is for access to those negotiated peers. However, just because a customer is paying indirectly for peer access doesn't establish an obligation on the ISP to support arbitrary uncompensated resource consumption on the behalf of their contracted peers. It's an entirely separate contract.

Cogent should pay Comcast market peering costs for the unreciprocated bandwidth they send to Comcast. That's what this whole dispute is about. Arguing that Comcast's refusal to build out and maintain significant infrastructure to support a peering imbalance imposed on them by a peering partner via abuse of a good-faith concession, at their own expense, is somehow "taking hostages" is just silly. Cogent oversold.

I think that Comcast sucks as much as the next guy, but I really don't think they're the bad guy here (as much as it pains me to say that). It looks that way to consumers because Comcast is the only contact affected users have with the whole industry (how many people who have been upset at crappy Netflix quality know who Cogent is, or what a transit agreement is?), but that doesn't mean that's actually an accurate perspective.


This is stupid, Cogent is providing Comcast customers with what they are requesting. Now Comcast says no way, I don't give a shit what my customers are asking for, you have to pay us as well. How is Cogent abusing Comcast by providing Comcast's customers with data they have requested.

No, I don't think Comcast is being abused at all and its much more likely they just don't give a shit what their customers want because they know there is nothing the customer can do about it and so they let service degrade in order to extort more money.


Because Cogent and Comcast have a peering agreement in which they agree to exchange traffic that is more or less symmetric (since that means that both partners will be spending approximately equivalent amounts on infrastructure development and maintenance). It has been customary for consumer-bound traffic to not count against the balance, since it was a relatively minor portion of the overall traffic exchanged. With the advent of high-resolution consumer-oriented video services like Netflix, that has changed - consumer traffic has become a significant portion of overall traffic exchanged, which causes a flow disparity between the two peering partners and places a disproportionate burden on one.

If this were not consumer-bound data, then Cogent would likely already be paying for the disparity. As it is, it's a good guess that Cogent has refused to compensate Comcast for the additional load on their network build-out and maintenance costs, instead relying on the good-faith consumer data exception to exempt them from paying for it. Comcast's response to this was to steal Netflix's business from Cogent, thereby alleviating the uncompensated load that Cogent was placing on their network and improving Netflix's deliverability.

There's no such thing as a free lunch. If you're a service provider, you're paying someone for delivery of your bytes. Netflix just cut out the middleman here. It's not like Netflix didn't already have to pay for the bytes they delivered - they just changed who they're paying.


I thought Comcast had to follow the net neutrality rules until 2018 regardless of that court decision a few weeks ago... Is this NFLX caving because nobody is enforcing the rules Comcast agreed to? (Edited for englishy)


The net neutrality rules were written to correct/prevent the abuses that were discovered around 2005. Since then ISPs have invented new loopholes (like targeted non-upgrading) that technically aren't covered by the old net neutrality rules.


In addition to the absurd, outsized cost, I don't want to pay Comcast et al. -- directly or indirectly -- these extorted prices, because they turn around and spend from these enormous profit margins to further constrain and "prioritize" my use.

The only solution, is to cut off this excess profit that has become the lifeblood of this systematic abuse.

And where these oligopolies have essentially captured the market to the point of excluding any and all competition: End them, legally. Break them up. Specifically legalize and perhaps even foster municipal broadband.

There may be no ultimate, static solution. But there are definitely next steps, and they are overdue.


Cogent has had peering disputes not only with consumer ISPs like Comcast, but with nearly every other tier one provider out there. Cogent is known to provide very cheap bandwidth but on the flip side they are also known to not be a mutually beneficial partner for settlement free peering. If you have cogent as a provider you should know this. Netflix knows that and thus has offered to provide settlement to Comcast to get better peering rather than be in the overcrowded Cogent link.


So it begins. The end of the neutral internet.

I am already paying Comcast for access to the internet! Why does Netflix have to pay on top it?

That said, I have Netflix and Comcast and I have never experienced any issues.

Edit: Actually I realize, this is an agreement to allow Netflix to colocate some of their servers at Comcast hubs. So it is not actually that bad.


> The end of the neutral internet.

"Neutral" is a vague term that in this circumstance conveys no actual meaning.

Settlement free peering is the exception, not the rule. Almost everyone pays for access to the Internet, whether it's buying transit or peering with settlement.

Nothing new to see here, move along.


It's only a problem for Americans, not the entire Internet.


Unsaid among all this - Netflix could have simply paid a higher quality transit provider (Level 3) to provide them transit, and there wouldn't have been any problems (Level 3 maintains some headroom on their peering circuits).

The reality is that it was likely cheaper for Netflix to just pay for peering with Comcast directly.


They have, and guess what? The same shit happened. Its almost like everywhere they go Comcast ends up doing this, kind of like they are specifically targeting Netflix.


"They have, and guess what? The same shit happened."

What are you referring to? Comcast and Netflix just started peering a few days ago.


Excuse me if I'm wrong but I think the parent might be referring to a 2010 dispute between Comcast and Level 3 regarding settlement free peering (a lot of the direct Level 3 or Comcast links seem dead now but some summary articles I can find are [1] and [2]). From these summary articles, the dispute happened after Level 3 agreed to become a CDN for Netflix.

One tangent that I don't know where else to ask is: why would an ISP agree to peer with Netflix, if Netflix weren't paying them. Reading about openconnect makes it seem that Netflix wants free peering.

My understanding of the situation is as follows. 1) Settlement free peering agreement are equivalent to free peering agreements. 2) Companies agree to these if they send roughly the same amount of traffic in both directions. So I'll agree to peer with you if I want to send ~N GB of traffic through your network and you want to send ~N GB through mine. 3) (and this is where I'm really unsure what I'm talking about) Anyone who peers with Netflix deals with handling traffic from Netflix to it's customers. But what traffic goes the other way? What traffic would an ISP send through Netflix network to make the relationship mutually beneficial?

Isn't it just the normal way of things that content producers pay ISP to send traffic through their networks?

Thanks for any clarifications!

[1]http://www.telecompetitor.com/level-3comcast-dispute-revives... and

[2]http://www.telecompetitor.com/behind-the-level-3-comcast-pee...


No ISP would agree settlement free peering with Netflix - it has always been the case that Netflix has to pay people. They chose to go with the cheapest provider, Cogent, and Cogent did not have the capacity to provide the services that Netflix required. So Netflix started looking elsewhere, and presumably the next cheapest deal they could get was by peering directly with Comcast.

Your understanding is absolutely correct - this is not a Net Neutrality issue, but a Peering Dispute, plain and simple.


So if that is the case, when I read the Netflix openconnect page at [1], the list of ISPs there do NOT engage in free peering? I was assuming they were because of the second sentence on that page: "ISPs can do this either by free peering...".

Also, another tangent: I'm wondering what happens when Netflix installs one of their cache servers at an ISP data center. If 10 customers want to stream House of Cards and that's 10GB of data, I'm assuming Netflix has to end up paying somewhere for 10GB of upload bandwidth.

But if the Netflix cache server was in a nearby ISP data center, would Netflix only pay for 1GB of bandwidth (to upload to their servers), and not pay for 10GB of bandwidth as the customers hit the cache server? I'm assuming that the ISP makes some of it back by charging Netflix to place their servers locally, but it's still cheaper then paying to stream all the bandwidth.

If an ISP were completely focused on maximizing it's profit and not on what Netflix quality was to it's customer, does the ISP have any incentive to install Netflix cache server?

Thanks again for clearing up these questions!

[1]https://signup.netflix.com/openconnect


When Netflix installs their cache servers at ISP data centers, they purchase transit from those data centers and pay for it at the 95th percentile like any other data center (or CDN) customer of that ISP. Netflix pays for the traffic they put onto the network.

The ISP has a big incentive to have Netflix increase their quality wherever there is strong competition for Broadband Internet. The more competition, the more incentive.

Here is an excellent article on transit: http://blog.streamingmedia.com/2014/02/transit-works-costs-i...


I think what I'm seeing is that Netflix doesn't really care about neutrality beyond its ability to enable their business. Which is basically what we as consumers should have assumed from the start.


This is a terrible precedent to set, and if it's true, I'm very disappointed in Netflix. Comcast is vulnerable right now since it's seeking to merge with TWC. Now's the time to pull out the big guns and show Congress, the FTC, and the citizens of the US why net neutrality is important. But instead of taking advantage, they cave and even pay off Comcast in what's essentially extortion. A disturbing vision of the future broken Internet to come.


"The gravy-train has ended." -Major ISPs

So much for net neutrality. Now ISPs have another way to make more $$. Keep those profits higher - push smaller streaming services out.


Wait wait wait...so, a company that has something like 60-70% of the market in total monopoly now has successfully ransomed another company who depends on their services?

Motherfuckers.


Where are you getting those numbers? It would seem [1] that Comcast has about 24% broadband market share. Subtracting the 3 million or so planned divested lines, even a post-merger Comcast would only have about 34% of the market -- which may be too high for comfort, but is nowhere near 60-70%

[1] http://gigaom2.files.wordpress.com/2013/05/usbroadbandsubscr...


Each coverage area is a separate market, and AFAIK cable companies have 60-70% share within their areas (mostly because cable is so much less bad than DSL, so the market is working somewhat).


Is there a non pay-walled version available?

This may mark the end of net neutrality.



Here's a direct link using Google as the referrer: http://www.google.com/#q=Netflix+Agrees+to+Pay+Comcast+to+En...


This worked for me, thanks!


http://online.wsj.com/news/articles/SB1000142405270230483470...

I think links with Google as a referrer show you the whole article.


http://bit.ly/NoS35l Just google the headline, they allow access to requests with google referrers.


Not working. Does this only work on non-mobile devices?


it works for me. are you sure you are not filtering out your referer header somehow?


Link without paywall: http://www.google.com/#q=Netflix+Agrees to+Pay+Comcast+to+End+Traffic+Jam&btnI


For purposes of easy clicking, here's the same link without the space: http://www.google.com/#q=Netflix+Agrees+to+Pay+Comcast+to+En...


Since when is it /#q= instead of /?q=? This doesn't work for people with noscript I think.


The ?q= doesn't cause the referrer to get set properly.


Oh thanks, I didn't know that!


Heres what I would be trying if I was netflix. Probably sure I'd have trouble pushing it against the rest of the company but I'd try this:

1) Collect evidence of network tampering/filtering

2) Get agreement in place to pay for better service

3) Collect the rest of the evidence that network filtering was in place when it gets lifted.

4) Simultaneously hit comcast up for breaching the neutrality agreements, while notifying all comcast customers that comcast is forcing netflix to charge its customers more for access.. that it might have to be along to them to pay... but we are working on ways yo prevent it.

Try and squeeze comcast from both ends at once and public embarrass them, while trying to keep the higher ground, looking like the peoples hero, under dog etc...

I doubt many companies would take that risk, but that what I'd try.


They have already done 1 and 2. The problem is #4 isn't possible because there aren't neutrality agreements to breach. Comcast provides shit service to Netflix and that's perfectly legal.


I have been waiting 4 years to switch to Sonic Fusion, but I am too far from the CO. I wish there was some way to take local action to help them expand to my area. I know many people would be interested.


I refuse to play this game - I'm back to torrenting.


Or better yet, let someone else do the work for you:

http://watch32.com

I'm curious to see how this plays out because the endgame for the internet is free-fast-failsafe. I wonder how long it will take for companies to litigate themselves into irrelevance. I actually feel kind of bad for netflix in this case, because when you make a deal with the devil, he always wins, and I hate to see them go someday :-(


> Or better yet, let someone else do the work for you:

> http://watch32.com

Unfortunately, this (and most other similar) sites use Flash, something I'm trying to avoid[0]. They also tend to have only low-quality, grainy video (not that Netflix is much better even when buffering isn't an issue).

There is also the issue of these sites/videos coming and going. If they host the files on 3rd party platforms (like this site is doing), then once the site is popular those videos quickly get taken down. Then you're stuck clicking through trying to find that one site (if there's even one left) that still hasn't taken down the video. OTOH, if the files are self-hosted, then when the site becomes popular, it can no longer support the bandwidth costs, and goes under.

Torrent sites, however, have lasted for years. I've been using the same private torrent site for nearly a decade. It's reliable and fast, and it has a wide selection of content, both old and new. And I never have to deal with Flash.

0: https://news.ycombinator.com/item?id=7287151



This deal doesn't seem as bad as many make it out to be. The agreement appears to consist of a an interconnection arrangement between Comcast's backbone and Netflix's streaming POPs.

Prior to the agreement, the traffic was routed through Cogent, and Comcast didn't want to increase the interconnection bandwidth with Cogent.

I don't see this as a blow to Net Neutrality, assuming the deal is as described.


For everyone in Europe:

Monday evening there will be a vote on net neutrality. CONTACT YOUR REPRESENTATIVE

http://www.savetheinternet.eu/

(dutch) https://www.bof.nl/2014/02/21/maandag-d-day-voor-netneutrali...

Explain your concerns to them.


Hard to underestimate the ramifications of an agreement like this. Many would say this is the market acting in the way it should.


I'd say my internet provider is fucking me over in failing to provide me with reliable connectivity I'm paying for. But that's just me.


Yeah, that sounds like the market I'm familiar with.


Local Cable companies are not a free competitive market; they are state-sanctioned local monopolies.


Actually they are natural monopolies.


The cabling is a natural monopoly. Who is allowed to service you over it is not. Thats why many countries have some degree of unbundling of service and provision of wires, eg I can get many providers of ADSL here over the same phone lines that are owned by a monopoly provider. Alas not for fibre though yet (UK).


Why would Verizon want to spend 10 million wiring up your neighborhood and potentially not get you as a customer in an area where Comcast has already spent 10 million to do so? Also why should Comcast be forced to let competitors use their infrastructure they've spent massive amounts of money on building and maintaining?


Little bit from my sibling response and a little bit more from the fact that they drained the ratepayers, government (through tax credits and accelerated depreciation), and the USF to the tune of $200 billion to provide universal broadband access (which they didn't do.)

We paid for the network, anybody selling transit across it should be able to use it.


Who agreed to pay for what, where?


If service is provided over Verizon or Comcast lines, they would still get paid for that. If they provide internet over their lines, they would get paid more. (Unless they operate their wholesale system like AT&T did, with wholesale access prices above retail prices including internet... grumble)


I only became aware that you can get Fibre from other providers this week:

https://news.ycombinator.com/item?id=7271128


Not really. The unbundling is restricted (you have to use BT equipment) and there is no price cap that BT can charge. See http://www.theregister.co.uk/2010/06/02/bt_fibre_eu/ from the time. Eventually we will get full LLU. Ofcom does claim http://media.ofcom.org.uk/2013/04/25/uk-broadband-competitio... there are 80 providers of fibre, but the terms are not attractive that I can find. I dont mind paying a bit more for non BT but not if its capped and has the same congestion as BT as its on their equipment...


Can't they be both?


With a monopolist ISP? How are they arguing that one?


Comcast Profit Jumps 28.6% on Growth of Broadband: http://www.nytimes.com/2013/08/01/business/media/comcast-pro...



Glad I don't have to deal with Comcast but I'm canceling Netflix if this is true.


One way to make this ugly real fast is for content providers like Netflix/Amazon to completely block subscribers from ISPs that apply QoS to negatively impact them.

Say something like "we can't provide a great experience so we think you should have no experience at all because it wouldn't be fair with us and with you. we are sorry." and make it explicit that it's because of the ISP they are using.

The ISP think they are selling gold when they are more likely selling a commodity. Sure, at some locations it's indeed like gold, but how much of that is artificially enforced?

These pipes aren't worth much if there is no content flowing through them.


So your expectation is that if Netflix intentionally prevents users from using their service because they don't like the user's choice of ISP, that will turn out well for Netflix?


If you want to ignore all the details in this discussion, how much power each company has and how much they're willing to leverage it.. sure, go ahead.

The whole net neutrality discussion is a power play between content providers and fiber owners, nothing else. The discussion about freedom and competition is awesome (and right!), but at the end it's these two huge groups fighting each other while we are screaming "think of the children!".

What Netflix would be losing if a subset of their customers on provider X is already getting crappy service and it can't do anything about that? At some point they will have to decide what to do about all those complaints and refund requests.


What would Netflix be losing? A lot of customers? And handing all of them over to Comcast's on-demand service. It'd be a massive win for Comcast.

There's no net neutrality issue in this discussion at all. Comcast is simply declining to upgrade its peering with Cogent. How is that net neutrality? If Netflix comes to me tomorrow and decides to buy all their transit through me, is Comcast somehow required to give me lots of free peering?

Netflix might be the majority(?) of Cogent's traffic, and Comcast might decide that it's not worth upgrading the connections for them. But as long as Comcast isn't discriminating traffic on their end (via some traffic shaping mechanism, I'd guess), then it's simply a matter of "Comcast thinks Cogent has enough connectivity, and sees no reason for them to upgrade the connections while getting nothing in return".

Their customers may disagree, but it up to them to not buy the service. (Perhaps the real problem here is lack of competition in the ISP market?)


Taking only this episode into account, you're absolutely right.

I was commenting on net neutrality issues and that was obviously not related here.


For those of you not famaliar with how peering agreements work, usually somebody will get one of the backbone providers to give them bandwidth in exchange for connecting the backbone provider to other peers. It's ideal when there is an equal exchange here and no money is involved. Netflix wants something different: they want to peer directly with Comcast/TWC/Verizon but not provide any network in exchange. For such an arrangement, it is fair that a network would charge them for this access.


You guys need to understand that the slow streaming that netflix had problems with is actually netflix fault. They choose to use the cheapest peer possible(cogent). So them paying comcast to be a direct peer makes complete sense as comcast will not be sending 1/1000th of the traffic it is recieving on these peering connections.

This has nothing to do with you having to pay more to stream other services. It has to do with netflix finally deciding to stop placing blame and further their own product.


I think Netflix should've just announced to it's customers that Comcast was throttling their network speeds before doing this. It is the same way that current companies in the US ask you to write to your legislatures about unfair policy changes, the most recent one I can think of in California is the public/private ban they're trying to pass on electronic cigarettes which I believe is still going on.


Given Netflix and Youtube account for rougly 50% of peak time traffic, I wonder if Google will also follow suit.

I'm all for net neutrality, but I think Netflix should've been paying some usage fee into a communal fund that is off-limit to ISPs, but can be used to improve broadband quality for everyone.


Google owns a lot of backbone so they can do a peering agreement with Comcast/Level 3 which would be mutually beneficial. Netflix has nothing to offer in terms of peering, so their attempt at getting similar terms is just a negotiating tactic.


Paid peering is frowned upon, but it happens all the time. This is not a threat to net neutrality.


I talked to a Comcast Loyalty Agent 2 days ago about my troubles keeping Netflix going on my devices and was told that Comcast has "superior" Netflix performance and if I am having any trouble it certainly isn't Comcast.


We need to do something guys. Probably this falls under some monopoly law or practice against free market because it hurts way too much all the current (and future) Netflix competition.


What I don't understand is why Comcast refused to put NetFlix's CDN gear into their network/DC setup. That would solve the problem...


How would you feel about someone putting their equipment on your private network? What if the equipment they install is low quality? who gets the blame? Who is in charge of QOS?

It is not as simple as you think


Will the telecommunications industry ever become competitive? The barrier of entry seems just too high with the amount of infrastructure it takes.


Only america can you charge twice for the same product


I've got a site. It has large files. Is Comcast required to provide free peering for me? Where do I sign up for my 10GbE port (my site's small so 10G is enough)?

Oh wait, I buy from a friend who runs an ISP. Turns out, he only has a single 10G connection for all his customers. Can he call Comcast for a free peering arrangement now?

That's not how things work.


This is anti-competitive. It is clearly unconstitutional. I hope the FCC still has a chance to compensate for their shortcomings.


Net neutrality is dead. The FCC has failed us.


That was their whole point, isn't it? To extort a toll for going over the bridge. A a classic trolling method.


There is another way: http://www.open-ix.org/


While many make reference to the mythical creature Net Neutrality, I think at best this will be mostly a way for Comcast to enjoy some time for extra revenue and perhaps a chance to develop a closer partnership with a company like Netflix. They could use the extra revenue to improve their network and the relationship to give them a fighting chance when local access monopolies are broken through new access options.


They could use the money to improve their network...but likely won't.

I recall the federal government providing billions to the cable companies in the 90s for high speed broadband that never materialized.


"Comcast is getting paid by Netflix? I want to be paid by Netflix too!" -every other ISP


I want to see Comcast try to throttle Google, Facebook, et al. and see how it ends up in court.


The weirdest shit is that people are okay with it, at least enough to not do anything about it.


I just cancelled my netflix subscription. I hope you do the same. Let them know why.


I did, about a week ago, due to poor picture quality and a library barely better than Amazon Prime, which I pay for anyway for the shipping. Had they made this Comcast deal a few weeks ago I probably would have kept them if the picture quality were better, but now that I've cancelled I won't bother to sign up again to see if it improves.

I was surprised that when I cancelled I didn't even get a single form asking why.

Netflix is $8 a month for all you can stream, so I suppose I can't expect any more than a poor library of B movies and old TV shows.

It's not up to me how Netflix and Comcast sort out their business disputes. Netflix was not delivering service that was worth their measly $8 a month.


Any link where we don't have to login after reading the first three lines?


I am getting tired of those news websites requiring you to register and login to read their content


Well now every other ISP is considering doing what Comcast just did.


this. is. awesome!

hollywood pressures comcast.

comcast shakedown netflix.

netflix offers money to comcast, comcast and hollywood is pleased.

so far it sucks... but then:

netflix pass that as tax to user.

users flee from comcast.

new providers appears.

hollywood cant shake them down.

for the first time, ISP in america have a truly competitive market.

the end.


Haven't they shot themselves (and everyone else!) in the foot?


I'm going to start my own ISP. Anyone want to invest?


I'm sold, nutjob123.


They just killed the internet.


NOOOO! They took you hostage netflix and you gave in! Only bad will come from this...


I guess they had to pick between Lawyers vs Comcast.


Did they just break the internet?


Oh this doesn't bode well.


Goodbye Net Neutrality :(


Capitalism wins again!


Wow. Net neutrality is dead. Here comes Google fiber (with it's own selective routing...)

I'm thinking of switching to Sonic.net - they seem to respect Net neutrality. Does anyone have Sonic.net experience pro or con ??


Hooray for the free and open market!! It's not extortion, it's capitalistic innovation!!




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: