I'm not too versed in this argument, so could someone please explain why Netflix's activities aren't covered by the normal peering agreements that make the "internet backbone" work? Doesn't Netflix pay it's ISP appropriately which gets disseminated by the other ISPs the traffic gets routed through?
Netflix doesn't have a "primary ISP", they are a network. They would like to freely peer to networks and ISPs that they can physically reach (e.g.: US ISPs), and they are happy to pay carriers for transit, that is bringing their traffic elsewhere in the world where they don't have a physical presence.
Most large ISPs won't freely peer with Netflix though; that's actually an industry standard, because peering is usually regulated by some kind of traffic balance between packets entering and exiting the network; e.g.: network A will peer with network B is they both exchange similar amounts of traffic in both directions. In Netflix case, it's obvious that this is impossible.
In those cases, Netflix would rely on paying the transit service; that is, they pay Cogent to bring their traffic into Comcast, because Comcast and Cogent already have some (secret) agreement in place (either feee or paid peer, who knows); Netflix would be happy to just pay Cogent to reach Comcast users. They would call it "a compromise" because their best option would be to freely peer with Comcast, but again, it's been an industry standard forever that peering is not only a factor of physical connection, but also of commercial power and quantity of packets sent and received (who needs to send more is the one that has to pay).
Netflix is now saying that Comcast is deliberately letting their peering with Cogent saturate and refusing to upgrade it (or offering economic conditions that Cogent won't accept, which is the same). This way, Netflix must enter a direct (paid) peering agreement with Comcast, to be able to offer good quality to Comcast users; in that agreement, Comcast will let Netflix pay whatever they see fit to reach their users. The peering here would obviously be paid and not free, since (again) there is a strong disproportion between packets going in/out the network.
If Netflix were to make their users' devices upload as much data as they download (even junk data which would be discarded at by their edge routers), would they solve the problem of the in/out data imbalance, and thereby be able to take advantage of free peering (with no bandwidth cap) with Comcast?
I suspect this wouldn't be the case, and the outcome would still be about their bargaining position and the negotiation process.
Can anyone with knowledge of these types of negotiations comment?
Well, that would be cheating, and Comcast would refuse to be cheated. But let's talk of a hypothetical P2pflix that is a video service with almost balanced up/down packets generated by end users.
For p2pflix, such negotiations would be far easier. The reason is that Comcast would need to somehow send those packets into p2pdflix network (they can't flat out refuse to deliver them in a net neutrality environment, or people would complain that the Internet is broken) so the alternative would be to pay a carrier that then peers with p2pflix. So instead of both paying their carriers, they might come into an agreement to do a free peering.
Peering is exactly that: when network A realizes that it's paying carrier C1 too much for traffic going into network B, they call B on the phone and ask for a free peering; network B checks how much they're paying carrier C2 to bring traffic into network A, and afterwards agrees to peer for a certain amount (e.g.: 4Gbps). They choose a data center where they both have a presence, setup a router with 4Gbps routing power, and reconfigure their BGP routes. Done, and the bills with their respective carriers go down, and everybody is happy.
If there is some disproportion, a paid peering might be negotiated; one of the two network would still be paying, but stil a lot less. Say network A is sending 4Gbps into network B, and network B is sending 6Gbps into network A. They can either free peer for 4Gbps (plus B would balance the extra 2Gbps through a carrier to reach A), or B might agree to pay directly A to have up to 6Gbps of peering, paying only the extra 2Gbps. Even assuming that the price per gig is the same of the carrier (and it's usually lower), it's still a good deal for both parts.
The problems happen with YouTube and Netflix because they don't have this negotiation power with ISPs, because they have a very big monodirectional flow of traffic. Let's also remember that on top of being monodirectional, the absolute amount of traffic is staggering: together, they make something like 45% of the TOTAL US TRAFFIC.
Netflix says that Comcast is letting their peering with Cogent saturate, and that's probably true, but I wouldn't be surprised if Comcast measured that 90% of that peering is being used for Netflix traffic. Otherwise, they wouldn't let it saturate because not only they would disrupt Netflix traffic for their users, but also any other website traffic that happens to be delivered through Cogent. (Well actually I guess Comcast doesn't really care that much about their quality of service to end users, but you get my point).
To me, it's not a clear cut. Obviously Netflix is a good and nice company providing a disrupting service we all love, and Comcast is a big bad ISP with monopolistic control on access in most zones and doing big profits with shitty service. So well, it's kind of easy to side, but I wouldn't say that Comcast's requests of being paid directly by Netflix is totally unreasonable, given the way the Internet has always worked up until now.
* Netflix cannot make money if individual Internet users cannot connect to Netflix and stream their movies. Netflix brings in a lot of revenue from its customers, and even after the cost of its own Internet connection manages to turn a handsome profit.
* ISPs stand between Netflix and their users. Many ISPs, such as Comcast, are also involved in the business of selling subscriptions to TV channels (usually with plans that require people to pay for hundreds of channels they do not watch) -- a business that is dying because of Internet streaming. Since Netflix relies on ISPs to have customers at all, ISPs want to demand money from Netflix (as well as from their customers).
* Somewhere in there are the actual users of Netflix, who have no particular say in any of this. Most of those users would have been happier with BitTorrent, but are too terrified of lawsuits (and even if they did turn to torrents, certain ISPs -- i.e. Comcast -- have been known to deliberately interfere with their customers' traffic to thwart torrents).
If your reading of this is, "ISPs are greedy and want to double-dip," well, yeah, that's what is happening here.
When the traffic flow was Netflix->Cogent->Comcast, peering didn't really come into it because both Netflix and Comcast were transit customers of Cogent. When Comcast is a customer, they get to decide how much pipe to pay for, and in this case they decided to pay for not nearly enough bandwidth to handle their customers' demand for Netflix.
The Netflix article that's also on the HN front page right now has a good diagram showing how they work with all other ISPs vs. how Comcast insists that they work with Comcast:
