Small example: an actual company I do some work for is in the business of delivering creative assets to distributors. This results in an egress of around 180TB per month, which is, on average just, around 500Mb/s.

So, this company currently operates 2 racks in commercial data centers, linked via 10Gb/s Ethernet-over-DWDM, with 2x512Mb/s and 1x1Gb/s Internet uplinks per DC. Each rack has 2 generic-OEM servers with ~64 AMD Zen cores, 1/2TB RAM, ~8TB NVMe and ~100TB SAS RAID6 storage per node.

Just the cost-savings over egress on AWS is enough to justify that setup, including the cost of an engineer to keep it all up and running (even though the effort required for that turns out to be minimal).

So, are cloud providers ignoring a significant market here, or is the markup on their current customers lucrative enough?

My theory: it forces third party services into the same cloud.

Suppose you use AWS and you want to pay a third party SaaS provider for some service involving moderate-to-large amounts of data. Here’s one of many examples:

https://www.snowflake.com/en/data-cloud/pricing-options/

And look at this remarkable choice: you get to pick AWS, Azure, or GCP! Snowflake is paying a lot of money to host on those clouds, and they’re passing those costs on to customers.

Snowflake is big. They have lots of engineers. They are obviously cloud-agnostic: they already support three clouds. It would surely be much cheaper to operate a physical facility, and they could plausibly offer better performance (because NVMe is amazing), and they could split the cost savings with customers. But they don’t, and my theory is that egress from customers to Snowflake would negate any cost savings, and the variable nature of the costs would scare away customers.

So my theory is that the ways that customers avoid egress fees makes the major clouds a lot of money. IMO regulators should take a very careful look at this, but it’s an excellent business decision on the parts of the clouds.

Everyone knew what the costs were going in.

AWS, cloud was NEVER the cheaper solution. It was never faster, it was never better.

It was easier, and easier came with a price. Like lambs to the slaughter lots of people embraced it. Amazon is a profitable company because of it.

[1]: https://www.agriculturedive.com/news/koch-foods-antitrust-ch...

[2]: https://www.computerweekly.com/news/366566360/Googles-data-e...

[3]: https://blog.min.io/googles-new-egress-policy/

I've not thought about the unintended consequences of this, but it feels like a reasonable regulation to have.

The asymmetry is obviously meant to trap customers, which is anti-competitive.

Additionally, settlement free peering is usually based on having a roughly balanced traffic ratio (something like 2:1 or 3:1 counts as roughly balanced), attracting more ingress traffic by making it free vs having paid egress helps the provider balance their ratios so they may help their case for settlement free peering.

I kind of doubt it's easier. If you use your own servers you need some stuff to manage them. If you use cloud, you need some cloud engineers to manage your cloud infrastructure. On top of that, if your developers use cloud APIs and frameworks, they have to learn that.

1: We have a dozen or so regular clients and being in a relatively high salary cost country, while our cloud bill isn't insignificant it's still far less than one full time employee, even less so than having 24/7 on-call handled (we have large enough 24/7 critical clients).

2: The cloud provides broken down billing, it's easy to pass it to the customer and just point to it, having our own employee we would need to structure up the billing process somehow. Our customers are outside of the IT field and doesn't care as long as it works and doesn't cost too much (it's a mostly fixed expenditure and easily separable from our development hours).

3: We have one client that does on-prem (partly out of habit) that's actually looking at parts to the cloud, partly because they've expanded internationally but I wouldn't be surprised if it's a bit of testing the waters since they have to spend a fair bit of their time on security and stability (security doesn't go away entirely of course but I'd be more sure of my backup processes on a large cloud compared to just locally).

4: I'm a tad on the fence of cloud-api's:

On the positive side: No need to implement a bunch of systems(badly?) and no need to build/install monitoring tools for those (and the extra administration burden that comes with updating those tools for X number of clients).

On the negative: There's a tad of churn on some cloud-api's, luckily Azure is mostly stable with plenty of migration time until you come around to a customer again (reading about Goog's churn feels like horror stories), and yes there is a bit of vendor lock-in if you go all-out (luckily we've not seen any reasons to change over).

I used to provide services like that, on call, and the number of ops related interventions for my cloud customers were consistently higher than for those on colo setups.

I think the bread and butter for the big clouds however are customers like the shop I'm at (see sibling comment), we can do development work and put it on our clients (mostly non-IT that has no interesting in hiring their own department) and if we for some reason need to part ways it's easy to "hand-over-the-keys" since subscriptions can be handed over between organizations.

But getting customers back is usually equally easy since there was never any fuzz when leaving so when they've become dissatisfied with the "cheap" options they don't remember leaving being a problem, regardless the winner is the cloud provider (but the costs are still small enough for each client that nobody is unhappy).

It's like virtual lego - the bricks fit the same way as physical lego but the difference is you can rent as many virtual bricks as you like, almost instantly with little up-front cost.

You still need to know how to build the lego pirate ship either way.

I disagree. The company that put their infrastructure on AWS may well have made an informed choice. But the third party service hosted in AWS doesn’t have a choice, because AWS would punish their mutual customer with egress fees if the service moved out of AWS.

I don't think making an informed decision has anything to do with competitiveness in any case. Example I might make an informed decision to publish my app on the app store, but this doesn't mean its practices are not anti-competition.

Cloud wasn't predatory in this... the price has always been there. If you lease a car and they mandate you get your oil changed with them for a price, and the price sucks, that's YOUR fault.

No one got locked into the cloud and THEN the egress prices went up... everyone went in knowing this.

The industry needs to man up and own shitty decisions rather than double down on them forever.

I'd argue it's still quite predatory. Basically, they reel in users with other services/products and then charge extortionate prices for egress specifically with massive margins just because they can.

In certain use cases it becomes all or nothing, either you host everything on the cloud or nothing at all which is certainly abusive and highly anti-competitive.

No one hid the prices from any one... it was all very up front and out in the open. No more data centers no more systems admins, no more capacity planing just scale on a dime... Here is the price chart....

And everyone didn't want to do that HARD work any more. They could just grow and grow and grow and it would save them money.

You know what happens when your not gonna run out of storage or bandwidth or servers... Everything gets bloated.

SO sure, we can say abusive... Amazon abused all the poor stupid vp's who took the lazy way out and let their systems get so fat and bloated on button mashing "l" shaped engineers. Crying about the lock in, about the pricing after you signed up for the service is your own fault. Take the hit and move on!

Knowing the cost beforehand and anti-competitive are two different things.

Just another reason so many orgs are getting heartburn from going too deep too fast into the cloud.

  Cloudflare  --  Free for most services
  OVH Cloud  --  Free and unlimited
  Scaleway  --  Free for most services

  Hetzner  20-60 TB / mo per instance  $1.08

  Linode  1-20 TB / mo per instance  $5.00
  Oracle Cloud  10 TB / mo  $8.50

  Backblaze  3x the amount of data stored  $10.00
  Bunny CDN  --  $10.00
  DigitalOcean  100 GB - 10 TB / mo per instance  $10.00
  UpCloud  500 GB - 24 TB / mo per instance  $10.77
  Vultr  2 TB / mo for most services  $10.00

  Fly.io  100 GB / mo  $20.00

  Microsoft Azure  100 GB / mo  $78.30
  Amazon Web Services  100 GB / mo  $92.16
  Railway  --  $100.00
  Zeabur  10-100 GB, depends on plan  $100.00
  Google Cloud  Depends on service  $111.60

  Render  100 GB - 1 TB, depends on plan  $300.00
  Vercel  100 GB - 1 TB, depends on plan  $400.00
  Netlify  100 GB - 1 TB, depends on plan  $550.00

I'm starting to think of cloud as less of an asset and as more of a liability. We can leverage them for temporary scale, but in no way will we tie ourselves to a particular vendor.

The adage is that you should outsource the things that are not your core competence. However if your product is infrastructure for developers then outsourcing that is going to be painful.

You are forced to pass down costs from your upstream supplier, and headroom is necessary (somewhat) because your provider can dictate terms essentially on a whim.

I feel like this is such an obvious statement, but we do seem to be in the gold-rush of avoiding responsibility so I'm not certain.

That’s a very expensive proposition. While the paper cost of a few racks in a colo is low - the people time to get where you want to be is high. If you mess up in this evolution - there is a risk that others outcompete you on something that isn’t your core product.

I find this is mostly an issue with a lot of shops simply having nobody with actual ops experience.

The problem comes when you then wish to have things like CI/CD, distributed databases etc.

And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it.

Done this many times.

You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sales ...; COMMIT TRANSACTION" (so all product prices and ... are included in the transaction "and nothing goes wrong"). It does not scale, it does not scale on postgres, it does not scale on Aurora, it does not scale on Spanner, it does not scale anywhere. It will scale on MySQL and MariaDB because it will just ignore your SELECT statements, which will cause a disaster at some point.

2) The big problem with cloud databases is the same as with normal databases: let bad developers just do what they want, and you'll have 40 queries per pageload, each returning 20-30 Mb of data. You NEED to optimize as soon as you hit even medium scale. And, sorry, "just slap a cache in front of it" is very likely to make the problem worse.

Cloud Hyperscalers salivate seeing this, have simply said: "don't pay an annoying database/system admin to tell you to FIX IT, simply pay us 10x what you pay that guy".

There's 2 kinds of people. There's people who look at the $10 pageload as both a technical embarassement and a potential business disaster. And there's business people who build hyperscalers, dine the CTO and hold a competition for a $100 pageload, while ordering their next Ferrari.

3) There's nothing automatic about automatic sharding. And if you "just shard UUID", your query performance will be limited. It will not scale. It just won't, you need to THINK about WHAT you shard, and HOW. No exceptions.

4) you'd be surprised how far ansible + k8s + postgres operator goes (for both CI/CD and distributed databases)

Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud.

Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort.

Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste.

2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%.

By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI.

Given this backdrop, I never worked on colos again (this all went down 10+ years ago).

Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed.

1. Don't do migrations at once. In a large deployment you likely have multiple locations anyway, so do one at a time, and/or rack by rack.

2. Verify usage levels, and take advantage of your ability to scale into cloud instances to ensure you only provision tiny headroom on your non-cloud environment. Doing so makes the cost gap to a cloud environment much bigger.

3. The advantage is being able to tune servers to your workload. That you had to fight for SSDs is already a big red flag - if you don't have a culture of testing and picking based on what actually work, yeah, you may actually be best off in the cloud because you'll overpay by a massive factor no matter what you choose and in a cloud you can at least rapidly rectify the repeated major mistakes that will be made.

4. Don't pay upfront unless you're at a scale where it brings you discounts at a return that makes it worth it. Rent or lease to own. For smaller environments, rent managed servers. The cloud provider dropping prices 2 years in shouldn't matter because by year 3 you should have paid off the hardware at an average cost well below anyway, and start cycling out hardware as your load requires and as/when hardware fails and/or it's not cost effective to have it take up space in your racks any more. Depending on your setup, that might be soon, or you might find yourself still running 6+ year old hardware you paid off 3 years ago for many tasks.

If you can't beat cloud on TCO, you're doing something very wrong. I use cloud for lots of things that aren't cost sensitive, but there are extremely few scenarios where it is the cost effective option.

On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I recall correctly, but we were in the realm of "we will have to hire people to maintain this project at 24/7 uptime". As the people time was relatively fixed to manage a sane on-call rotation - shrinking the infra footprint would have simply increased the people time portion of the TCO. If the footprint had shrunk by 75% - the maintenance cost would have become problematic. Due to the higher intensity work of build outs vs. maintenance, the people time factor would have risen had we started in more experimental amounts. As it happened, tech salaries also rose ~2x over the 3-5 years that Colo existed.

On the workload front, as happens with many large organizations - there are teams who have the attitude of "don't touch my stuff". At the time that the buy was initiated, there were several estimates indicating that we would need 4x the hardware we were buying in 5 years time.

Whenever someone says that they are beating cloud TCO, I'd suggest you do the following math.

- Sum up all costs related to the on-prem hardware, include smart-hands, power, Racks/network gear, power adaptors.

- Calculate a depreciation schedule for the existing hardware targeted to 3 years.

- Add a 10% cost of capital to depreciate future savings

- Amortize all "non-server" hardware costs onto your principal bottleneck (be it CPU/Network/Memory/Storage)

- Project how your savings compare against cloud costs under different assumptions of future cloud discounts. Both negotiated and public.

- Project how your savings compare under different utilization assumptions.

- Add a bus factor into the investment, what happens if your team leaves/hardware gets smashed/Workload becomes more efficient/Other crises occurs

Next, and more controversially

- Add in the people cost of everyone involved in the maintenance of your on-prem infrastructure.

- Project what happens under varying assumptions for how much of a raise they will ask for. What happens if tech compensation rises by another 2x over the next 3 years?

- Explore what happens if on cloud you needed X% fewer engineers. Depending on what you are doing X could be 0%, or even a negative percentage - but for many shops some fraction of existing work can be automated or avoided.

It was this math that made me turn away from non-cloud offerings. I still use cloud alternatives for some personal projects - but I don't bill myself in that setting.

Hence my caveat, but racks are typically rented at a fixed price per rack, and most places never even reach a scale where they need a full rack per location, so my experience is that for most people it pays to keep servers quite a bit longer because most people have spare space in racks that are already being paid for.

Once you're at a scale where you typically will have whole racks aging out at once, it shifts the calculation somewhat, but then to it really varies greatly depending on e.g. your balance between storage and compute. It's very rare it pays to throw out hardware on the 3 year mark, except in markets where the power and real-estate cost is unusually high, but then moving your hosting wholesale often pays off - e.g. I've in the past moved entirely workloads from London to cheaper locations.

> Otherwise, we did mortgage the servers in the backend - buying upfront gave us around a 50% discount on top of the steep discounts you get from buying direct from an integrator vs. Dell et. al.

Either you loan-financed the hardware or the earlier mention of the "use of VC funds" was irrelevant, then. And nobody gives you a 50% discount for paying up front. You might have paid 50% less than the full cost of the purchase price + interest rate, sure. That's not a discount, that's not having to pay interest. In the end, whether you buy on credit, rent, or lease to own, you either way get a cost-curve per server per month, and that is what is relevant to plug into your models.

> On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I recall correctly

Migration is a one off, so this only makes sense given a time frame to write it off over. 10%-20% written off over 1-2 years wouldn't be completely crazy, though high. If your time horizon is so short that this matters to you, then you have organizational problems.

Put another way: While I did consulting in this space, I'd often offer to do the transition for clients for a percentage of their savings for the first few months, because I knew exactly how much these transitions would take us, and the clients would take a look at the proposals and accept my hourly rate instead when they realised how much they'd save, how quickly.

Reduction in egress fees alone often paid for my fees in couple of months (one system I migrated, which was admittedly atypical, saw hosting costs drop 90% thanks to egress fees alone), and we usually saw devops costs drop at the same time. Most of my clients outsourced 100% of their devops so the costs were easy to quantify.

> If the footprint had shrunk by 75% - the maintenance cost would have become problematic.

This is backward thinking. If the footprint drops by 75%, the cost for that drops. If your maintenance costs don't drop as fast, it doesn't matter - your total cost is still lower, but if your maintenance cost isn't elastic, you have an organizational problem.

And while whether or not the proportion spent on each factor then changes might be a political consideration, but if you then surrender savings because the budget "would have become problematic" then it's no wonder you ended up with a failed transition - if there are incentives to avoid savings to maintain numbers that were broken from the outset. This sounds more and more dysfunctional to me.

> Whenever someone says that they are beating cloud TCO, I'd suggest you do the following math.

Done all of these many times, and never once had cloud come out remotely competitive.

To your "controversial" point, what I usually see when people think their cloud setup is comparative is that they carry out no accounting of how much time they actually spend maintaining cloud-specific things. When they get to the point of handing it over to someone specializing in it (as I did for years), it's often a surprise to them just how much time they offload from their teams.

A few of the other things that seem to shine through here is 1) an assumption of capital outlays. Not needed capital outlays for coloed environments in any setup I've done in the last 20 years - did it for a handful before that; the cost of financing directly with the provider or via a leasing company is priced in when I compare costs with cloud because otherwise it wouldn't comparable. If you then want to pay upfront, that's a choice, not a necessity unless your credit is absolutely worthless.

2) Comparing only colo vs. cloud instead of adding in hybrid or managed hosting. If you build a pure self-honest environment a lot of the price advantage gets eaten up because you need to assume a far higher amount of spare capacity, which will drive up your cost even if you get it right, but people often end up far too conservative here and assume peaks far higher than what they ever need.

The moment you have a hybrid setup that can scale into cloud instances as needed, which is typically little extra effort to set up (you're going to be using an orchestrator anyway), you can easily double (or triple, if people were being conservative) the typical load on your colo servers, and cut the hardware and rack cost accordingly, and usually when people do this they still end up hardly ever actually spinning up cloud instances, because most peoples traffic varies far less than they'd like to think.

Even more so given there are now plenty of providers that offer you a seamless transition from colo, via managed server, to vps's to cloud instances, with often surprisingly little difference in time to spin up extra capacity. Your setup just has a method to register a new resource anyway, irrespective of what is underneath - I've deployed systems that way for nearly 20 years, now, with hybrid setups spanning the gamut from colo, via managed server, VPSs and AWS instances in a single setup.

The net effect tends to be to have to defend retaining the capability to scale into cloud because of how rarely it ends up being used.

3) an assumption that you need to hire people vs. e.g. outsourcing. Most companies never reach a scale where they need even a single full-time person doing hands-on ops - you're better off leaning on colo support, and retainers for monitoring and out-of-hours support for fractional scaling until you reach a scale where staffing several dozen full-time staff becomes viable. I've never had a problem scaling this up/down on an hour-by-hour basis with commitments for base-level needs on a month-by-month basis. For years I used to provide fractional support for companies to facilitate this type of thing.

4) an assumption that you can't automate the same things in a colo environment as in a cloud environment. For a well-managed colo environment, past racking hardware, if you can't boot the system via IPMI etc. straight into an image that automatically enrolls the server in your orchestrator, you're doing something wrong. If your cost of physically managing your servers are more than a rounding error, you''re doing it wrong.

Yet, the flexibility of public cloud environments is like the gold ticket for people doing devops consulting - when I was consulting in this space, the one constant was that the clients in cloud environments ended up paying me 2x-3x as much for assistance for similar size and complexity workloads. And I still usually cut their costs significantly compared to what they used to pay. E.g. the time that goes to maintaining network setups in a typical cloud setup that's solved by plugging things physically into isolated switches is staggering. Yes, usually people could do it cheaper than they are in public cloud setups, but the risk of getting it wrong also tends to be far higher.

The choice is not between "do all hardware and building" and "cloud". There's a whole spectrum in between. In fact, it's quite hard to do the building. You can go colo (building + power + electrical done by vendor, optionally network), dedicated (servers done by vendor), hybrid hosting, hyperscaler (you get VMs), ...

As soon as you move above colo, you don't have the hardware issues mentioned here. Or, well, you do have issues, but you "just" file a ticket.

Cloud is great for many things, but not cutting cost.

Especially since one does not have to backhaul the traffic to one specific PoP.

Cloudflare manages to run their Anycast CDN as a loss leader just fine.

Also, on the plan routing is atrocious, with all requests from Africa and Asia being directed into Europe, which helps keep costs down, but it cannot then be compared with fly.io.

Add on the usual and egregious $150-400/mo for a SMF/OS1 crossconnect in many datacenters.

Now, I'll grant you, "not even close" and handsome margins likely still applies but "I'm going to hook a server directly into a 1Gbps IP transit port for $200" vs. all the complexity of a hyperscaler network is not apples to apples.

Some of that is capex, and some is opex. But it's worth noting that the hyperscalers are doing something far, for more complex than almost any single-tenant physical installation would do.

In a hyperscaler cloud, I can run an instance anywhere in the AZ, and I can connect it to my VPC, and I get what appears to be, functions like, and performs as if I'm plugged in to conventional network infrastructure. This is amazing, and the clouds don't even charge for this service as such. There are papers written about how this works, presentations are given, costs are bragged about, and entire industries of software-defined networking are devoted to enabling use cases like this.

But an on-prem or normal datacenter installation doesn't need any of this. Your favorite networking equipment vendor will happily sell you a single switch with tens of Tbps of switching bankwidth and individual links that are pushing 1Tbps. It takes maybe 2RU. If you are serving 1 billion DAU, you can serve up a respectable amount of content on that one switch. You could stick all the content in one rack (good luck at that scale), or you can scale across a few racks, and you don't need any of the amazing things that hyperscalers use.

And, realistically, very very few companies will ever need to scale past that in a single datacenter -- there aren't a whole lot of places were even 100% market penetration can find 1 billion people all close enough to a single datacenter that it makes sense.

So the apples-to-oranges comparison cuts both ways. You really can get a lot of mileage out of just plugging some off-the-shelf hardware into a transit port or two.

All the other points you made are also excellent.

https://aws.amazon.com/lightsail/pricing/

If I were running a service like Snowflake, I would certainly appreciate the effortless scaling that the major clouds offer. But I also know what I pay for actual servers and roughly what I would pay in a major cloud, and I much prefer the former.

Anyway, there is an explicit egress charge: https://docs.snowflake.com/en/user-guide/cost-understanding-...

just with DataBricks. you don't pay when you don't use their software, cool. but then you pay for per core per second for the license in addition to running the VMs, right?

sure S3-like storage is 'cheap', but if you want up-to-date data pipelines you'll end up running these things for many hours each day. and plus you pay for each bucket operation, and so on.

This is hilarious. Now that nobody makes on prem software anymore and nobody knows how to do IT, it’s time to turn the screws. The prices are only going up. You can’t move your data because it’s all locked into proprietary silos and only readable by one app running in one place.

I told you so. So did loads of others. Oh well.

There was also a global policy rolled out which pushed the VPN client to various infra machines in branch offices. Some of them were running GitHub runners and data processing software which had terabytes of egress per month.

The whole thing was a disaster and they rolled it back within a week.

Cisco anyconnect VPN capable appliances (that can do 10GBE) are very expensive, and licenses are per user- so if an appliance needed an upgrade it is conceivable that it could cost $50k in the first year.

Pricing was around 75k for 1000 seats for a year. They thought it was going to be $25k a year but it turned out to be that a month.

• The cost of two hardware gateways (the depreciation cost, that is)

• The cost of 24x7xNBD (next business day) hardware support

• The cost of the user/session licenses (which might be depreciated also)

• The cost of software support/upgrades

But that extra $65k/mo to fully utilize the link is utterly and completely nuts. My mind boggles when someone calls that a good deal. I can buy and fully depreciate the equipment needed to utilize that in a couple of days. (Well, I can’t actually buy equipment on the AWS end, but I can _rent_ it, from AWS, for a lot less money than $64k/mo.)

And I don’t believe at all that it costs AWS anything close to this much to operate the thing. I can, after all, send 10Gbps between two EC2 instances (or between S3 and EC2) for a lot less money.

That $65k is simply a punitive charge IMO. AWS wants me to avoid paying it by not doing this or, in cases where I can’t avoid it (e.g. the thing is actually a link to my office or a factory or whatever) to collect as much money as they can without driving me off AWS entirely.

https://cloud.google.com/network-connectivity/docs/interconn...

Ctrl+F "Data transfer from a VPC network through a Cloud Interconnect connection"

There's a reason the favorite chart to exemplify value Cloudflare reps like to show is Argo Smart Routing, and why it costs about $100 per TB just like AWS and GCP.

The existence of the world wide web has tricked us into thinking that sending traffic anywhere is/should be the same, but of course it is not. So while the price you (a cloud customer) pay for egress pricing is (often) indiscriminate on where that traffic is going, using common sense, we can understand that some traffic is more expensive than others, and the price we pay is a blended price with that aspect "baked in" or "priced in".

But I would not use it for static assets. For this we use Bunny edge storage to provide faster response times at very reasonable prices.

The problem is that there are now multiple generations of software engineers that do not know how bandwidth is priced. They've only used managed providers that charge per unit of ingress/egress, at some fractional dollar per GB.

I show them bare metal providers and colo that bills by size of pipe rather than transfer. They refuse to believe it or assume there must be a catch. There usually isn’t, though sometimes the very cheapest skimp on things like rich peering and can be slightly slower or less reliable. But still cheapest is relative here. Expensive bare metal or colo bandwidth is still usually hundreds of times less than big three cloud egress.

It’s just nuts.

It’s a subset of a wider problem of multiple generations of developers being fully brainwashed by “cloud native” in lots of ways. What an amazing racket this all has been for providers…

$0.09/GB? I guess that’s just what it costs. Expected when you’ve never looked at, considered, or even heard of things like peering and buying transit or even co-location. Enter 95th percentile billing on a gig port for $500/mo or whatever…

Same goes for hardware. Want to watch something glorious? Take someone who has only ever used VMs, etc in big cloud and give them even a moderate spec new bare metal server. The performance melts their brains.

Then they realize the entire thing - colo, servers, bandwidth, even staff is a fraction of the cost.

The industry really found a way to stop the downward cost spiral of personal computing and bring everyone back into mainframe. Moore’s Law is not dead but you no longer see the benefits. The difference is just pocketed by cloud providers who will use the gains to pack more low performing VMs onto less hardware.

5c/GB is less than the 9c that Amazon charges, but still, ouch.

You pay hurricane for the bandwidth.

You buy space in Equinix DCs because they are network neutral and have a high density of ISP POPs, i.e you can light up fibre between your cage and HE, Level3, whatever other transits you want to use + peer with everyone on the local peering fabric super easily.

I consider Equinix transit to be an option only when you really can't be assed to do anything else and you don't intend to hammer the links anyway, otherwise yeah you should be doing all the things above because otherwise you could have rented colo in some shitty backwater DC instead.

No 1 believes they can do anything anymore. It's not just hosting and hardware either. Everything single thing becomes a SAAS that gets purchased and they all refuse to believe it can be done in-house.

> It’s just nuts.

It's exactly that. I've seen upper management in many companies value external solutions as more cost effective because if they had to do it internally it'd cost 2x as much.

It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers.

I think there is also some geo-specific pricing that gets hidden in a global price; bandwidth can be a lot more expensive in some locations than others and if you are charged 5x for egress in south america, nobody will use the south america locations and that's not good for business.

You mean like EC2 T series burstable CPU credits?

Then, they have the audacity to recommend you make everything Multi-AZ. “Don’t worry, RDS-RDS traffic isn’t billed!” Yes, but the apps running in us-east-1a that are hitting -1b and -1c due to load balancers most assuredly do bill for traffic.

I guess that’s just one more reason to prefer AWS over Azure.

https://cloud.google.com/exit-cloud

Also it could be possible to set up some hybrid solution and offload the egress heavy assets serving to another provider and only run the "brains" inside AWS/etc.

What's cheaper with cloud is that you don't need upfront costs, but here, you already have the whole infrastructure already there, up and running.

Why would you even consider getting rid of it and replacing it with AWS, makes no sense.

(You're wrong, there are many such scenarios)

People use AWS for the same reason they rent houses: less investment required and you can change things more quickly.

Yes, exactly. And it's very often the case that these capabilities are more valuable to users than "real price" as you mean. You... understand this, right? That value is measured contextually, not objectively?

We have another service that has 20x the bandwidth, but it's a legacy GAE that has super cheap premium egress... But I'm told that AWS says that their pre-pay discount will be competitive in about a year, the rate that we're growing.

The negotiated egress prices are much lower, so long as you are buying future egress. If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

Shouldn't it be the other way around? Why are they giving you are discount if you're not likely to jump?

Aka as long as you are not using the cloud as cloud the cloud can be cheap?

>rent the space and pay for upkeep for less than $4k a month?

I mean, you can just rent servers somewhere else.. or do a colocation rack. Also I would strongly suggest going with the first one

Egress bandwidth pricing has been the sacred cow money firehose forever, despite transit costs continuing to shrink.

see reddit, see amazon/aws outages taking with them netflix/disney plus etc

honestly its a lot better to keep your architecture cloud agnostics and test restores regulary on a different provider/region

also: store your backups somewhere else, not on your primary cloud provider, not connected to your auth system, not overwritable from anywhere

Of course, these come with increased cost, but I am thinking web retail on a large scale, or airline companies, where for example a downtime of a few hours will easily wipe out any savings made by relying on a local data center. It might not be the solution for a smaller company.

And if netflix/disney/slack/ring amazon themself can not do it, with a multiple with the ressources i have access to, good luck to you.

there are also times where your company might rely on a single aws product, and it might simply break an integration(aws iot mqtt ->sqs for example) took them 2 days to fix

The charge is not based on cost in the case of the big names. They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free.

Hence the high costs to deter this behavior.

This isn't remotely true.

The bandwidth alliance exists, and a lot of cloud companies are on the list: https://www.cloudflare.com/en-gb/bandwidth-alliance/

The actual answer is much more complicated. For example, Google Cloud offers two different bandwidth tiers: premium and standard. The calculation on the OP assumes premium since that's the default option, but obviously it's much more expensive.

Google cloud's "premium" bandwidth is much akin to AWS Global Accelerator since it utilizes Google's own backbone network for as long as possible before exiting at the nearest peering point between Google and whatever ISP your end user is at. AWS Global Accelerator has some other options available, that make it fundamentally a different product, but the routing characteristics are much more similar to GCP Premium bandwidth than anything else AWS offers.

This is why egress is dirt cheap on other platforms outside the big 3 cloud providers.

This is also why ingress is free.

With an exception to OVH, none of the cheap providers the article has listed have any kind of backbone network. They all rely fully on transit providers. Turns out backbone networks are expensive to operate!

OVH is the sole example of a provider that has a backbone network, and admittedly, it's pretty good. However, nowhere near expansive as the big three, and it falls flat in Asia (which is the hardest to route traffic in). Also OVH has to build datacenters so cheaply that one of them burnt to the ground in recent years...

(Cloudflare has a backbone too, but you have to pay a lot extra to use it. Linode uses the Akamai backbone now but that's a very recent acquisition and it's expected that Akamai will eventually raise costs significantly)

Yes, bandwidth is way too expensive on cloud providers. AWS Lightsail is proof of that. However, I see no reason to believe that this is purely for vendor locking, and nobody has been able to give any evidence of causation between the two beyond "well it's so expensive!!!"

It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform.

They therefore charge high fees to move your data off the platform to discourage this behavior. Meaning you now need to use cloud provider X’s services to do anything with the data.

Attempts at vendor lock-in have been core to software service companies since they were born.

Right but if this were the case then why does the Bandwidth Alliance allow you to move data at a much lower cost for 2/3 of the major cloud providers? If they _really_ cared so much about not allowing you to do processing with a third party, the Bandwidth Alliance wouldn't exist!

AWS is the sole hold-out here, and I think the way that Cloudflare worded this makes it pretty clear that the Bandwidth Alliance is basically a middle finger to AWS than anything else, but it also seems clear that the cloud companies aren't actively trying to make it costly to do data processing on a third party.

In fact if you want to move off GCP right now, Google will waive all egress fees to do so: https://cloud.google.com/blog/products/networking/eliminatin...

That isn't how business works. Companies maximize their profits and "balance" isn't a profit center. If it didn't benefit them in some customer leveraging way, they would charge for ingress.

You pay for everything. Either directly or indirectly. Indirectly often turns out to be much more expensive.

What I'm referring to is the practice of balancing peering ratios. That is, when you make transit/peering arrangements with other ISPs, some ISPs will charge more if the amount of data you're sending to them vs the amount of data you're receiving from them is not balanced. It is in Google's best financial interest to at least try to balance their pipes in this way.

Nothing in your comment rejects or disproves the claim that egress costs are vendor lockdown.

Your link to the bandwidth alliance explicitly states that their justification for network costs is unloading infrastructure costs onto end users as data fees. That's their only and best justification. This is clearly a business decision that has no bearing in operational costs.

Some cloud providers charge nothing, others only start charging after hitting a high threshold from a single instance. Do they not operate infrastructure?

It's their business, it's their business model. Some restaurants charge you for a glass of tap water too. Let's not pretend they do it because of infrastructure costs.

Yes you do pay for the rest of their infrastructure when you rent servers from them...

I'm not saying that the fees aren't extremely overpriced. I know what a gigabit port costs. But saying it's to keep vendor locking is just not true, and nobody has suggested any actual proof of it being true.

Of course, non-premium tier is v4 only, and only available at some locations.

I didn't pick the hosting I'm working with, but GCP IPv6 for instances seems to work fine, other than it costs more?

I experienced this a couple of times with different webhosts (in the Netherlands) over the last three to four years. Very recently my monthly bill at one of them went up from around 3,- to around 18,-.

Whenever I read about 'egress costs' I usually laugh about that because I don't even understand it; my data is my data, why pay to get it off your hard drive!? But for some time I'm also seeing dark clouds for my own use cases. I don't find it hard to believe anymore that webhost after webhost is consolidating, trying to play AWS, upping the prices a lot, and finally... also establishing egress costs!??

A 9x price hike in just a few years has got to be the least scrupulous PE move I’ve ever witnessed in NL. I agree with your prediction that it’s just a matter of time before they tack on exorbitant egress fees.

I think what you will discover is that network bandwidth is a finite resource you have to pay for and ration out to your customers, and that the easiest way to finitize and ration something your customers expect (errantly) to be infinite is to charge for it, and charge exactly that price that will cap demand!

Interesting. Which cloud providers do you have in mind?

Oracle has probably really good margins on egress costs. With AWS/GCP/Azure the costs are absurd because for a lot of their customers it's not a big cost during operation, but makes moving data off cost prohibitive. It's simply a vendor lock-in mechanism for them.

As a data company we really benefit from Scaleway, Hetzner and Ovh.

However, Scale way has no plans to add an US infra

And I don't know if Hetzner has US présence?

It's not like AWS or GCP have locations that are terribly far from major exchanges either.

> In North America, the median cross-connect price is $300, more than five times higher than $58, which is the median price in Europe.[1]

Of course, that difference means nothing to a hyperscaler, but it's led to a difference in pricing culture between the smaller providers.

1: https://www.fiercetelecom.com/telecom/study-u-s-fiber-cross-...

Except you paid the cost for the fiber connection. You also get one more 9 of reliability from their DC.

For example, AT&T fiber has no bandwidth caps and offers symmetric 1-10 gigabit service in an ever expanding number of areas throughout the US. I pay $55/mo for 1gbps service, and have done more than 100TB/mo for many months and it's fine.

Instead of relying on consumer ISP's to route gaming packets for speed, they would instead route them for costs. Taking extra hops and extra latency before reaching the target server.

Riot built an overlay network using their own fiber and negotiated BGP deals with consumer ISP's to get gaming traffic off the ISP's internal network on onto Riot's ASAP so they could route the packets directly to their gaming servers.

also, linked from that: https://technology.riotgames.com/news/fixing-internet-real-t...

A key issue discussed was some setting relating to BGP (or similar) that consumer ISPs wouldn't initially implement or wouldnt configure for Riot. Something related to weights or something that affected the cost analysis for routing decisions that prevented packets from wanting to take the Riot path.

edit:

Just fully read that QZ article. I think it does a good job of describing what they did with useful inforgraphics (esp the "Actual Data Route" graphic) for a layman audience. I think many of us here can extrapolate what kind of effort they had to go through to get that done.

Those are very different, the standard network is more akin to AWSs egress as GCPs Premium option (the default) uses local PoPs and dark fibre over Googles backbone to origin.

https://cloud.google.com/blog/products/networking/eliminatin...

This doesn’t change business as usual egress fees, it’s likely a stopgap to prevent being accused of anticompetitive practices.

It’s not as mature as the others, I don’t believe you can use your own encryption key for example, but why would anyone starting out not choose R2? Hopefully they push the market in their direction.

Typically this takes the form of creating "renditions" of the source video at lower resolutions, but you'll have to decode and re-encode the video to do that, hence transcoding.

Pretty much all video you watch online is delivered using an adaptive bitrate technology like HLS or DASH. That's what allows the video to automatically reduce its resolution if your network can't handle streaming the video without buffering.

If you're storing 1080p h265 MP4s in R2 and the client can only support 480p h264 it'll request that but you need to "appropriately transcode" for them.

The alternative is a service that does this for you, like Cloudflare Stream. Upload source video, service handles transcoding and integrates easily with standard video players with HLS or Dash.

Just in case: I am not OP

https://community.cloudflare.com/t/can-we-serve-video-with-r...

Please correct me if I am mistaken; also R2 is not a CDN, and more like s3 in terms of delivering from the edge

No issue with your comment specifically, just wondering if you know. currently using s3+cloudfront for mp4 storage+delivery, and would like to move to something better if possible.

  Unless you are an Enterprise customer, Cloudflare offers specific Paid Services (e.g., the Developer Platform, Images, and Stream) that you must use in order to serve video and other large files via the CDN

  The Cloudflare Developer Platform consists of the following Services: (i) Cloudflare Workers, a Service that permits developers to deploy and run encapsulated versions of their proprietary software source code (each a “Workers Script”) on Cloudflare’s edge servers; (ii) Cloudflare Pages, a JAMstack platform for frontend developers to collaborate and deploy websites; (iii) Cloudflare Queues, a managed message queuing service; (iv) Workers KV, D1, Durable Objects, Vectorize, Hyperdrive, and R2, storage offerings used to serve HTML and non-HTML content; and (v) Workers AI, a Service that allows customers to use Cloudflare’s inference infrastructure to invoke select third-party machine learning models (subject to applicable open source licenses or third-party terms of use for such models).

Yes, it uses Cloudflare's edge network and caching infrastructure when you use a custom domain (really the only option because usage of the default domain is very limited). So yes, it's a CDN :-)

https://community.cloudflare.com/t/cdn-support-for-r2/514573

> currently using s3+cloudfront for mp4 storage+delivery, and would like to move to something better if possible.

What I have found is that (A) the egress costs are extreme compared to R2 (remember, R2 does charge per read/write though) but (B) storage is cheaper on S3. You should do a study on your storage and egress, but unless you are storing way more than your egress TB, it's probably a good deal.

Cloudflare also has a mechanism for transparently pulling content from an upstream S3 compatible store into R2. I think it's called Super Slurper iirc

I think given the cost advantage for s3 for storage, it seems almost better to pull from R2 into s3 for long-term storage (some inverse slurper).

It's good to hear though that R2 can singlehandedly match s3+cloudfront; that being said, video delivery is a bit different I'd imagine, even s3+cloudfront is finnicky with range requests etc.

That, and it's usually HLS ABR, more about that on my parent comment.

This article is about egress costs for just operating a service that talks to the outside world, not just when migrating data to a new service.

But it makes sense because the counters get reset each month.

So if you consume "101TB" over a year, distributed as 90TB in January and then 1TB per month... You could get hit pretty hard in January and have free bandwidth for the rest of the year.

[0]: https://wasabi.com

So if you use 1TB storage but 2TB egress , it will not be free anymore.

Iirc eg Cloudflare does not allow videos etc. as part of their free tier.

edit: confused ingress and egress

There is some debate about the status of hosting video when using Cloudflare as a CDN.

These comparisons are for hosts/object store.

Cloudflare R2 has no restrictions on the type of content while not charging for egress.

HN thread: https://news.ycombinator.com/item?id=32242987

Cloudflare has limits on the size of objects that can be cached, making it relatively impractical to use for a download-focused site with resources more than a few hundred MB.

[1] https://blog.cloudflare.com/bandwidth-costs-around-the-world...

Databricks now has fully hosted too, but Snowflake still did not go the other way to introduce BYOC. Newer entrants, however - RedPanda, AnyScale, ClickHouse... - are going the way of providing both BYOC and pure SaaS options in their cloud. There are many other reasons to prefer BYOC, legal, security, data privacy, but network egress costs is one of them.

Perhaps on self-serve plans with insignificant traffic. On enterprise plans they do set bandwidth caps. Our company exceeded our bandwidth cap and our account rep reached out to reprice our contract.

I asked him if turning off Argo (a service billed by bandwidth) would remove the bandwidth overage and they said no.

TlDR: Cloudflare bandwidth isn’t free at the enterprise tier.