40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily.

If you also use containers you even get quite a bit of flexibility and agility.

To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on.

But then again, I'm not one of the cool kids...

Agreed. I hear some saying it's nice to deploy on a lambda because you don't need to know anything about the runtime environment. But it's never quite true. As you say, you do have to become an expert in all the intricacies of these proprietary deployment environments to get the best out of it and avoid getting burned. So an amount of effort has to be spent, anyway.

But the drawback is that now you spend this effort on learning what is after all a proprietary product of AWS. While AWS is massively popular, the knowledge doesn't translate to anywhere else so it locks you in. If you spent similar time learning the basics of Linux deployment and administration, your knowledge is lower level and more general.

But where it really gets you is when things go wrong and you need to dig deep to diagnose. Can I attach a debugger or watch socket traffic or run bpftrace or do any kind of diagnostics at all on that lambda instance? Oh sorry no, good luck.

Unless your budget is so low you can't afford a $5 VPS and your traffic is so low that your lambda bill will never reach $5, you're really better off deploying on a VPS that you control and can debug. Keep it simple.

Exactly, I learned Linux in late 90s. There are a few new softwares like nginx instead of Apache but I can bring up a new VPS and set it up using essentially same knowledge from late 90s. With a CDN and caching, you really do have a massively scalable service.

Don't get me wrong, I love Cloud but I will never use it for personal projects. It just feels like renting vs owning. My personal projects will have bugs, and I don't want to be on hook for thousands of dollars.

Learn Linux/Sysadmin once, and you can likely use that knowledge forever.

As for performance, one of my projects can sustain 300rps which is more than enough. Sure it cannot scale automatically to thousands and millions but I don't think many projects have that requirements.

On top of that, cloud tech can't be "set and forget" because of deprecation of apis and services.

Good platform-as-service can solve many things for you and let you focus on the core thing you are providing.

Obviously not everybody needs to worry so much about the stuff mentioned above. If you are providing SaaS solution, there’s a good chance some customer will start asking these questions as part of their procurement process.

Fixed workload, shared dev boxes, all kinds of small-scale operations fit very well onto a bare metal / VPS box in someone else's rack. The maintenance work often boils down to setting up cron with unattended upgrades and borgbackup, Docker with Traefik+ACME as the web frontend, etc.

PaaS solutions will often pull you in the direction of getting "addons" or using extra paid services for every component you might need: a relational database, a cache, a queue, a load balancer... Which are usually a very good call if you need to scale up, but which could've all been entries in your docker-compose.yml if that's all you needed.

Most importantly, and regardless of your hosting strategy, you can't just dismiss security, disaster recovery, or capacity planning. Clouds have outages and bottlenecks too, and no product can save you from human error.

Hey, that's not exactly a correct comparison. You don't simply get half of those in the cloud either. Log monitoring and disaster recovery is something you have to figure out yourself, the best clouds have are some foundations to build upon, and possibly - some cookiecutter template that might fit your use case (if you're really lucky it'll even be decent). And you can get same stuff on traditional servers, just with different pre-baked solutions (which also may or may not fit a particular use case and may vary from perfectly good to quite crappy).

People love to brag about all the features (most not needed for your casual website), but somehow no one tells the fact that those features just won't be there when you'll start to use the cloud - because you have to be actively aware that you need them, explicitly enable some, and explicitly spend time learning, setting up and testing others. Unless we're talking about PaaS (and not a "classic" cloud like AWS, GCP or Azure), you still need someone with some sysadmin experience - except that this person must wear a different kind of sweater (with $Cloud logo rather than Tux or Beastie).

All you get is some hardening an OS + managed software (like LB servers and databases) patching. Which is something that's not that hard to do on a self-managed server (well, the software updates part; hardening is a rabbit hole). But not application patching, mind you - that's your responsibility to maintain your app, the very best it can do is to run a security audit (which you can get as a service separately). And even though managed databases are tuned (still a lot of manual tuning to do if you want the engine to truly purr) and maintained they aren't all that fun and peaches the marketing materials say - sometimes you just have to e.g. spin up your own self-hosted PostgreSQL to perform the tricky migration, then replicate it back to a managed solution.

There are many hosts out there who will take care of _all_ of that for you. You just need to install your stuff on the box, and they take care of the rest. Logs and monitoring... you'll prolly want something custom anyway rather than whatever anyone else offers.

> figure how to implement data encryption at rest,

you have to do that anyway

> implement high-availability and so on

again, many hosts who provide easy ways to manage that for you.

Well, this isn't exactly the best practice to follow - but, yes, this is very common, and it works (until someone finds that there's a vulnerability to exploit). So, from a purely practical perspective - software updates are certainly not a hard requirement, one can do quite well without them (just be aware about the risks).

Anyway, there are many fairly age-tested solutions for updating software on the fly, live kernel patching and live network sockets included.

And a lot of cloud software updates (e.g. managed databases) incur noticeable downtime so you need a failover (and appropriate software design that can survive it). But then you can do the very same thing with two servers, no cloud magic necessary.

Also, out-of-box distributions those days can be said to be already "hardened" compared to what we had 15 years ago.

Much of this complexity I eliminate by using the Immutable Server pattern:

Specifically I deploy my app as a Docker container hosted on a virtual machine cluster managed by AWS ElasticBeanstalk. OS upgrades and patches (outside the container) are done by AWS.

If anything goes wrong with a VM I just terminate it and a different fresh VM spins up to take its place.

What do you do if you need to launch another instance of the app? For staging or testing.

In contrast I've ran some pretty hot-and-heavy MySQL/Postgres databases on basic DO/Linode VPS's for half of that cost with much success. I get these cloud tools give a ton of "out of box" features - but you are paying for it at the end of the day.

Anecdotally I've noticed a huge shift away from general devs/engineers having general CLI/Linux/sysadmin knowledge.

I say this as an engineer who has a ton of relational DB experience. Before cloud providers w/DB options were so plentiful devs/engineers/sysadmins would have to set this up by hand.

---

> If you had no knowledge on standing up a DB server

My point is, "but yea what if you do?" There's actual value in that I've found, especially when I'm looking to save pennies at the early-stage of a startup while in the customer aq phase. Many folks have expertise in those areas without needing a cloud provider.

I do get the "but all of these things!" argument (access control, patches, updates, monitoring, logging) but often those are easily solved/solvable problems even without leveraging cloud offerings. I can get very far along with an ELK/TIG stack + uptime robot and basic VPS networking features... all for a very affordable monthly bill on the infra.

If you or any of your developers don't know how to do that, you/they should seriously seriously spend the 40 minutes it takes to learn.

It’s not hard to imagine a dev being brought up 5 years ago into a React world, with JS in the backend and something like Vercel or Netlify to deploy their app.

Someone like that might be entirely uncomfortable when dumped into a plain old Linux server, and would definitely take longer than 40 minutes to even get their head wrapped around what’s going on.

That’s not a knock on them - they get their job done, and to the business that’s all that matters. But yeah, just trying to, I don’t even know, empathise? Try to understand? Who knows. It’s just a different world now.

If a grown up person (who is presumably making a good money in their React/JS thing) can't figure on how to install a web server in 40 minutes (with all SO, Google and a bazillion of articles on how to do so) then...

Like come on, twenty years ago you still needed to get your ass off the chair and go to the bookstore, find and buy a proper book and actually read it. Now you just need to type "Ubuntu 20 install webserver" in to the search and voila.

If your endpoint takes 10ms to run and you're running it constantly for a month, it will cost you less than $0.05 (it will actually be about half of that because you get the first 1M invocations free each month, but ignoring that...).

As you point out, running RDS is more expensive, but RDS isn't serverless. You would probably want to look at DynamoDB for a serverless database, which is considerably more affordable than RDS.

Moreover, the serverless stuff is considerably easier to configure and operate than a VM.

But less frequent invocations are actually an even better case for Lambda versus a VPS because it suggests less wasted time (yeah, you have to pay for cold starts, but with a VPS you're paying for all of that time your service is up irrespective of whether or not it's being used).

> 100's of milliseconds minimum. When your serverless endpoint needs to integrate with other AWS services, the configuration can get complicated fast. VPC endpoints, IAM roles, security groups... it goes on and on.

This is all true for a VPS as well. Assuming you care about reliability, you need to run multiple instances of your VPS behind a load balancer which implies a fair bit of networking. Moreover, you need to manage the hosts themselves, so configuring log aggregation, metrics collection, process management, SSH, deployment, hardening, etc, etc, etc.

Further still, your compute will need to communicate with the comparable services as in your lambda hypothetical, so you still need to deal with IAM and some more networking stuff. Maybe you'd say "gotcha! I would just run my databases on my VPS instances!" which is cool, but now you need to configure monitoring, backups, replication, failover, and so on for your databases versus using DynamoDB (and I would bet a lambda/dynamodb workload than the same workload on a reliable VPS/whatever-database stack).

Of course, if you're running a hobby blog or something that doesn't need reliability of scalability, then a $5 VPS is probably fine (maybe even an S3 bucket behind a CloudFront distribution, which would likely be free).

My big problem with lambda / serverless is the developer experience is pretty awful. The time between making a change, deploying, and seeing the result of that change is slow. You can work around this (with tools like localstack), but it's often not close enough to the real environment. You'll still waste tons of time debugging permissions issues when you do a real deploy.

> My big problem with lambda / serverless is the developer experience is pretty awful. The time between making a change, deploying, and seeing the result of that change is slow. You can work around this (with tools like localstack), but it's often not close enough to the real environment. You'll still waste tons of time debugging permissions issues when you do a real deploy.

I haven't had much of a problem. Once I figure out the shape of the inbound payload, it's pretty easy to test locally. I can't think of any reason the runtime environment would be an issue. Debugging IAM is tedious, but I just created one Terraform module that I use for all of my functions so I don't have to slog through the IAM stuff every time I make a function.

Yep - and to be 100% candid on my current project I'm running 100% with GCP Functions, AppEngine Task Queues, and Firestore for persistence. I'm also AWS certified and have worked as a DevOps engineer in a PCI-regulated environment leveraging things like DyanmoDB, Lambda, etc.

Cool part about running that way is my current billing is near zilch. Woo! Totally on-board with what you're saying... literally living it now =)

I'm just saying that there's a ton of ways to skin a cat. And, personally I wouldn't paint with as large of a brush to say "serverless stuff is considerably easier to configure and operate than a VM." A lot of times this rings true, but I anecdotally wouldn't say it is always the case.

I'm being pedantic, but I'm also trying to make a case that not always do cloud design patterns/"best-practices" make sense.

Not my experience. Our AWS lambda cost is close to $1K/month and the usage isn't that much so we could easily host this on a few (for redundancy) of the smallish VPS instances for far less money.

When you start thinking these, they are not so easy. (1) means you can’t just run “apt upgrade” on every server - you need to manage the updates to make sure they get tested first. (2) is kind of ok, but requires some work on regular basis (at least checking things) (3) means you need to monitor the updates for your stack and classify them. You can get feeds for example for Ubuntu, but does that cover whole stack. And checking these weekly (or daily) actually gets boring.

All this stuff can be done, but IMHO it is time consuming and takes time from more important things. The weekly/monthly JIRA tickets for checking x, y and z get quite annoying when you also have n other things to finish. Then you start slacking on them and feel the pain when trying to collect evidence for the next procurement process check.

If you have tens or hundreds of servers and can have a separate infra team with professional sysadmins then this is all fine. My rant is mainly for small teams, where same guys are supposed to develop and run things.

We disagree webserver logs contain 'hardly any PII'. You may find it insignificant PII, but it does not constitute 'hardly'. IMO its a circumstantial goldmine.

https://www.huntonprivacyblog.com/2009/07/10/washington-cour...

I am curious who "we" are- a royal plural?

We referred to you and me.

That is true.

> Good platform-as-service can solve many things for you and let you focus on the core thing you are providing.

I don't see how with platform-as-a-service solutions you can skip documentation, maintenance, log monitoring, disaster recovery, backups, test procedures, or encryption at rest.

Scalability should be the last thing in mind if your product sucks.

A lot of us work at big companies with lots of users and demand and aren't just making POCs, but every time we bring up scaling or maintenance or monitoring we are told not to worry about that yet.

This whole Tell HN is about personal projects where costs matter, the startup tangent here is just a jab at people underestimating the raw processing power of modern hardware.

Big corps can do what big corps do with their big corp money. 200 user startups don't necessarily have hate kind of money but pretend they will be big corp very soon and thus need to spend big corp money.

But then there will probably needs to be a devops team and careful planning how to architect infrastructure anyway.

Also in such cases cloud pricing becomes prohibitively expensive unless you get a special deal.

Essentially you just upload a ZIP of your application, and register a handler function that takes a JSON payload.

Obviously this is quite a bit more boring than a K8s cluster, with a bunch of nodes, networking, Helm charts, etc.

I would posit that even compared to something like a DO Droplet, Serverless is still kind of boring. Everything is going to fit into the model of registering a handler function to accept a JSON payload. There's no debate about whether we're going to have Nginx, or what USGI runtime we're using. It's just a function.

And with Serverless, your cost for doing a couple million, 2-second-long requests is about four cents.

It's not my kind of boring, although I sort of see what you're saying.

It presents a simple facade, but it's built on complex infrastructure you have no way to have any visibility into. So when things go wrong it's a nightmare.

To me boring tech means super simple, KISS to the extreme. Something I can diagnose fully when necessary without any layers of complexity (let alone proprietary complexity I can't access) standing in the way.

Here the OP kind of got caught by the terrible DX that is almost natural to serverless IMO.

This seems wrong to me ? Can you explain a bit more ? Are these just API requests or ?

The price is $0.0000166667 for every GB-second on X86. So it looks like I've also misplaced a decimal. It's $0.20 per 1M requests with 1GB of memory.

Lambdas can be sized from 128MB to 10GB, and pricing depends on the resources allotted.

Ultimately, this pricing just represents the compute time for a function to complete. That function can do whatever it wants.

There are additional costs to put an API Gateway in front of it, for instance, to make it a publicly accessible API.

[1] https://aws.amazon.com/blogs/aws/announcing-aws-lambda-funct...

So if I had a "hello world" function, that basically just returned a constant JSON payload...I'm seriously looking at a pittance, pennies for millions of requests? I.e. $0.20?

Millions of requests for a static file is easily done on a dedicated server in a few seconds if the file is small. $0.2 per request is very expensive.

To put things in to perspective: If you buy a very cheap new server for $1K, then you would be able to handle small static requests worth thousands for dollars each day according to AWS pricing. That's a nice return on investment!

It's $0.20 per request, without memory.

In my opinion, it is easier to switch from serverless pattern to VM instances, and much more difficult to switch from VM to serverless without a major rewrite.

This all comes while keeping the data on servers belonging to EU companies. Using Cloud servers is rather dodgy as shown by the fact Google Analytics, Google Fonts, etc are all illegal.

Everyone keeps talking about the ease of just getting a new server and not having to spend time repairing servers. This is an over sold problem. How often have we ever dealt with hardware problems 8-10 years ago? I only remember one server going down because of hardware issues that cause problems all the other times there were failover servers and the hardware issues were fixed by datacentre staff before anyone even noticed there was an issue in the first place. With tools like ansible and chef, you can provision a new server super quickly so even if the server gets a corrupt configure of whatever, you can just wipe the server and rebuild.

Most of us are not at the scale where we would realsitically see the benefits of cloud computing for hosting.

There was a thread here the other day on how DevOps has failed. And this should hopefully show everyone why DevOps is needed. Cloud infrastructure is complex and needs specialists, otherwise simple mistakes can be very costly. If I make a new lambda, you better believe I'm watching invocations.

I don't know, I'm curious about the actual requirements vs the marketing buy-in and resume building that may happen when designing the system.

I've consulted for companies that went all in on AWS Lambda + AWS SQS, only to transition them to an EC2 instance that performs the same computations at a fraction of the cost.

No — you do not need K8 on day one.

And typically with much lower latency/request times.

GCP functions, AWS Lambdas, etc... my anecdotal experience is that they are way slower for request times vs a $20 VPS running the exact same workload.

I assume you just dropped a shit ton of features on your move to that EC2 instance unless of course you wasted more time then it would take to get everything working better on Lambda/SQS.

Yes. People don't realize how little that is. Million requests per day is 12 requests per second, on average. Even if we factor in peak load at 20x that, it is still well within reach of exceptionally modest hardware, assuming at least vaguely sane software stack.

Often you get lucky and the cluster of managed services you select happen to scale along the metrics of your resource use. Many people's resource use patterns are similar and the managed service people take advantage of that. This is nice! But it's a trade for downside risk: you may find that your resource use patterns differ from the 90% case and your spend goes up very fast or your scaling hits a wall.

In my experience a lot of designing a service architecture is picking where you want your complexity. Services (in or out of containers) running on VMs have a simple billing and architectural model. In my experience they form a good basis to organize your other resources around and are a good foundation to grow from.

If you app has multi-dozen users, and needs to "scale" something is wrong.

Usually, the hard part of scaling isn't raw compute power, it's scaling your datastore after you've already exhausted the option of throwing more compute power at it, and this problem remains whether you're on the cloud or not.

Until you hit that problem however, throwing more hardware at it is the right solution (you may be surprised just how much load a single Postgres server on bare-metal can handle).

It isn't really that hard.

At a former startup where I worked as senior engineer, that was our original approach. Then one weekend we tripled our userbase and horizontally scaling required massive changes in database architecture, sharding solutions, etc.

"Just buy more servers and load balance" is the short painful route. Carefully planning out and taking advantage of scalable architecture that can be provided to you less expensively because it too, runs at scale is the hard method. The fact that its easy to shoot yourself in the foot with it doesn't make it the easy route.

You took the short easy route out of the gate instead of building for what you were planning to handle. This is the opposite of what I espouse. If you were going about it my way, you build that in from the ground up. Harder and a bit bumpier to roll out from the get go, but has worked well for me in the past.

I wouldn't call it "good" or "intelligent", but it's definitely hard.

I have a few dedicated servers monthly cost ~100$, never ran into any problems, for a brief period I needed to pay cloudflare for some dns management that was getting a bit heavy, but even that was because I didn't know enough as to how to optimize.

For any personal projects etc / POC's even startups launching to less than 100k daily users (traffic and load type dependant ofc) hold off on the AWS>

I also don't like the lambda architecture in AWS as it seems to lock you into it in a sense, you're almost tied to that infrastructure for good. Yes you can rework it but thats tech debt that may not be possible.

Love AWS but only as an enterprise solution.

> If you also use containers you even get quite a bit of flexibility and agility.

Yeah... and then the only difference is between a single host and multiple hosts. Guess which one Kubernetes is for?

With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems"

> can get away with not knowing basic sysadmin stuff

I promise you that the knowledge you need for your single dedicated server is also needed for k8s clusters, and I definitely can't imagine anyone who can maintain a k8s cluster but can't maintain a single linux host. It's more like that scaling horizontally comes with exponentially more difficult problems than what you're used to or have heard of.

Obviously the billing is an issue but that doesn't negate the whole concept. They should definitely implement better expense controls. A simple hard cap to activate if you want would fix this for good.

You have a script that needs to run, without fail, at a certain time every day.

You have an endpoint that is mission-critical and even a second of downtime would cause insane manual workload.

You have an endpoint that needs to have as little end-user latency as physically possible.

These are all real examples from my work in fintech. The solution to all of them is a few dollars in cloud functions in Lambda and Lambda@Edge. Now imagine having to provision, orchestrate and maintain dozens of dedicated hosts all by yourself, simply for these handful scripts. The expense of employee time alone would make it absolutely idiotic to go that route and if anything this would equal massive scale day - but even worse, because you didn't even plan for it beforehand.

I prefer aws over metal for these kind of setups, but for many other cases I definitely do not; just a dedi or a vps with docker or k8s and/or something like openfaas is enough for almost all startups and beyond. Making it literally impossible to make mistakes like OP. And when needed maybe failover or load balancing.

We also have a few dedicated servers, but mostly only for infrequently accessed data and logging that doesn’t need to be highly available.

I really can’t understand why this argument keeps coming up. Different solutions for different usecases. Yet anytime Kubernetes or cloud functions are discussed people come in and go like “hurr durr my single Hetzner dedicated server can do all of that and doesn’t have these problems”

I agree with you though on the case by case.

Assuming we're talking about side-projects/hobbyist development: when I'm doing that I want as few variable expenses as possible, and they usually don't require the purported benefits of the cloud. If such a project needs to scale: I'll bolt it onto a cloud at that point, or sprinkle in specific services to address the pain points.

[1]: https://tritondatacenter.com/blog/manta-postmortem-7-27-2015

I agree with this, but the choice of what other individuals play with is not ours to make. There's still countless of youngsters working on bare metal, for whatever it's worth, they just grow up in a world with a lot of existing abstractions now.

> Assuming we're talking about side-projects/hobbyist development: when I'm doing that I want as few variable expenses as possible, and they usually don't require the purported benefits of the cloud. If such a project needs to scale: I'll bolt it onto a cloud at that point, or sprinkle in specific services to address the pain points.

100%, but the choice is free for anyone themselves to make. OP is furthering his knowledge and experience in a very specialised and highly sought after sector. Maybe he'll come to the same conclusion at some point, or maybe he comes from bare metal and has different perspectives now and thinks that redundancy is always awesome.

Hetzner provides incredibly beefy machines for a good price.

I’ve been managing 2 machine cluster setup in the past using Hetzner and it worked fine.

Unfortunately I have had an incident with one machine going bad unexpectedly(RAM issue) and I was forced to spend whole day setting up a new one.

If you need the lowest possible latency then you should definetly do it yourself. Lambda, even with provisioned concurrency has a very high latency...

Your problems compound if you have to consider disaster recovery (DR) and need an offsite location you can failover to. Now you have to contend with data replication and all its concerns.

So even if your application only has a few dozen users things can get quite complex quite fast. AWS makes it simple, practically push-button. Deploying in multiple Availability Zones (AZ) is a no-brainer, and it isn't horribly difficult to deploy in multiple regions, complete with data replication. All while having no servers to configure and maintain. No containers to contend with, no Kubernetes to fight with. To me it's the most stupid-simple solution available today and it's ridiculously cheap! At least so long as you test before you deploy for world exposure!

Try doing this [0] with your dedicated servers in under 10 minutes.

And then try attaching highly-available scripts/cloud functions and dozens of different integrated functionality in seconds.

And then try setting up good Network ACL, firewalls, route tables, NAT gateways, load balancers with a few clicks.

... and people here are seriously suggesting that instead using AWS would equal a massive forced scaling operation. Lol.

[0]: https://i.ibb.co/TKmB9HX/image.png

Why does everyone assume everyone else works at tiny startups?

I've never worked at a startup, until last year I never worked at a company with less than 100k employees.

With all due respect, I don't think you've ever actually put together a local cluster.

A simple 4 machine k8s cluster sitting literally on dirt in my basement can scale out to the equivalent of thousands of dollars of AWS spend a month. I broke even on the initial purchase outlay for my workloads in less than a year.

The problems are almost never scaling the web servers. The problem is scaling the infrastructure that those servers need to be useful.

Generally - your DB is the first pain point, your network is the next.

If you can run it in a container, that service probably isn't the bottleneck for scaling, it's going to be whatever is providing the persistent disk for that service, and the network between the two.

Both of those things happen to also be fairly expensive to scale in the cloud as well.

I've used both comcast and google fiber while running my basement cluster. I think on google fiber I would have said production usage would be fine - I would pretty reliably get 1gbs down and ~800mbps up.

I wouldn't have redundant net, but the upload speeds are fine.

On Comcast... Comcast is really kind of a joke. Even on expensive business plans they cap upstream at 35mbps in my area (down is 1gbps) and the fastest way to get throttled to actually try using that upstream capacity.

Comcast does give me redundant failover to 5g by default now, which I guess is nice, but it's not worth much when I can't send a ton of data over the line anyways.

So point taken - if you're going to have to rent a room in a major metro for fiber access anyways, you probably want to be paying for cloud services (sadly - you'll pay a lot, since outbound traffic costs are high, and if 30mbps upstream isn't doing it for you, you're gonna blow through the free 100gb fast)

I have, and still do. It's neither redundant nor highly available. The power source isn't, the internet connection isn't and it's also located in my basement and not multiple regions.

> The problems are almost never scaling the web servers. The problem is scaling the infrastructure that those servers need to be useful.

There you go..

> Both of those things happen to also be fairly expensive to scale in the cloud as well.

No. [0]

[0]: https://i.ibb.co/TKmB9HX/image.png

My internal transfer costs are also.... drumroll... zero. (ok - technically, at some point I bought a 10gbs switch and 500ft of ethernet cable)

Lets talk about what it costs you put data back out onto the net, or into a different region. Then lets compare long term storage costs.

Because that shit is only cheap in the "hobbiest" range. Outside of that, outbound data and storage is fucking expensive in the cloud.

Really? In completely different physical locations with inter-region routing and NAT gateways for each region that route into a redundant load balancer? Then you're right and don't need AWS any more. However, I'd suspect your costs would not be very affordable either.

> Lets talk about what it costs you put data back out onto the net, or into a different region.

If you're seriously trying to compare your residential internet connection with something like this I'm not going to go there..

> Then lets compare long term storage costs.

Same logic applies.

I'm saying the things that are hard to do yourself aren't fucking cheap in the cloud either.

My counter point (and the one you keep dodging) is that the things that are EASY to do yourself aren't fucking cheap in the cloud, and they scale out just fine for the vast majority of non-Saas businesses.

> My counter point (and the one you keep dodging) is that the things that are EASY to do yourself aren't fucking cheap in the cloud, and they scale out just fine for the vast majority of non-Saas businesses.

And I don't even disagree with it for its inherent correctness, but because nobody ever said you absolutely have AWS for something that doesn't need it. Most users here tend to discard cloud platforms for even the most obvious and suitable use-cases, and keep on iterating how their single dedicated server can do all of it. I'm also not a big fan of telling someone they don't need to use AWS if it's clearly a side hobby project because that is how we all learned.. by playing with cool modern technology for fun.

The real issue is super simple.. AWS just needs a hard cap functionality so stuff like this can't happen.

I guess I don't really see the appeal of putting a hobby project into a space where a simple mistake can literally cost you thousands of dollars, with very little recourse.

I get to play with the vast majority of the fun tech - I just choose to do it on a k8s cluster I own, with extremely predictable costs, and frankly - more scaling capacity than I think most businesses need.

> Most users here tend to discard cloud platforms for even the most obvious and suitable use-cases, and keep on iterating how their single dedicated server can do all of it.

And this isn't really my stance - I'm absolutely not advocating for a single dedicated server (fuck - even my hobby instance is HA, 3 machines with a network lb, on a 5 hour UPS, with a redundant network. I personally don't care as much about the redundant network, but comcast provides it free to business customers in my area anyways through 5g equipment).

But I do think most (and I mean 75%+) businesses can spend 10k on a single server rack and go for literal years without problems.

If your business is not selling software as a service - don't buy into the bullshit marketing.

Even if your business is selling software as a service, think about it first. Small businesses tend to get sweetheart deals on costs in AWS, GCP, and Azure because their usage costs for those companies amortizes out to roughly zero, and because they know damn well that they'll end up paying through the nose if they make it to mid-size (or even just screw up once or twice - two fuck ups like the one here and you'd have bought yourself a decent server rack!)

For a lot of projects this doesn’t matter. The cost of downtime might be lower than the cost of high availability and occasional downtime is OK.

Most projects don't even actually suffer for modest downtime, although you can achieve AWS-comparable downtime even without AWS/Google/Azure-style "cloud" architecture.

It can make a huge difference in operating and engineering costs to know which bucket your project fits in. Good architectural foresight can even let you smoothly move from one to other if unexpected growth or a pivot indicate that it's warranted.

Engineering is about understanding the scope/scale of problems, not about being dogmatic or getting caught up in problems that don't apply.

Just curious - have you really researched cloud solutions or did you just compare the price of hosting EC2 instances in AWS vs having your own server? Because that's not what cloud is about.

And you are overpaying by a significant chunk. My raspberry pi - the old cheap one - can handle 10 million requests a day without breaking a sweat. If I push it, I can get up around 90 million requests a day without too much effort (it's only about 1 request/ms)

I really don't think most devs understand how fucking cheap hardware that's comparable to these services is.

Now - you might be using a host of other valuable features that your cloud provider gives you (things like edge servers near your customers, or truly significant outbound network traffic flows, or a very robust multi-region setup, or disk backing of some sort, etc).

But generally speaking - you ARE overpaying for cpu cycles in the cloud. It's not really up for debate.

There are certainly cases where AWS can be better (ease of edge networks and multi-region availability come to mind)

But outside of a very small set of cases (most of which are when companies are victims of their own success - which is actually a wonderful problem to have) what's the compelling reason to actually use AWS if "Of course I am paying more" for it?

In my opinion, if you can still run your db on a single machine - you don't fucking need the cloud yet. That covers a pretty large chunk of businesses.

Most of the "cloud" is convincing business that could self-host their entire stack on a raspberry pi in a basement that they should be spending thousands on cloud compute costs a year.

Fuck - the most damning evidence is simply how much fucking money these companies are making from upselling you cpu cycles that they're getting mostly for free. For the 4th quarter of 2021, amazon reported a profit of 5.2 billion on ~18 billion in revenue from AWS.

SQS is so good and that there isn't a great, easily deployed (sorry, RabbitMQ) option for exactly what it does is a real push towards AWS. (GCP Cloud PubSub is close enough, but I know AWS way better than I do GCP.) If I didn't need this, and I felt confident managing RabbitMQ as a queueing solution, I don't think AWS would be a compelling solution because $60 in Hetzner nodes, in a HA configuration, could do a lot of work.

(If this thing works I'll need an on-prem solution anyway, so I'll probably have to build that too--but that's "good problems to have".)

Totally agree on SQS, knowing PubSub as well I'd say they're pretty much on the same level. All the interconnectedness is where cloud platforms shine

We do have a hosted version of NATS called NGS that is multi-cloud, multi-geo and is really easy to set up https://synadia.com/ngs

And, you probably know a little bit about NATS, too, since, you know, you wrote it! :)

https://nats.io/support/

Plus the OP has already decided that free tiers don't work.

A single VPS is great if you don't need reliability or scale, but if you care about those things you probably need at least a few of them running behind load balancers, and you probably don't want humans manually poking at them (but rather you want some reproducibility). Moreover, there's a bunch of host management stuff you're taking upon yourself--configuring metrics collection and log collection and SSH and certs and stateful backups/replication/failover and application deployment and process management and a whole bunch of other things that come for free with serverless.

Implying that serverless/Kubernetes/whatever is just hype ("cool kids") while ignoring the pretty significant discrepancies in requirements is pretty silly.

However, if your statement is generally about CDNs such as Cloudflare or Bunny.net that don't have per request pricing, it makes sense.

CloudFront on the other hand will bleed you dry with their pricing.

If you have to pay by card and you have to subscribe always use a disposal virtual card. Most of the time these companies won't pursue you for payment because it's simply not worth the effort. The most likely outcome is that they'll just suspend your account. What you don't want happening is them taking money from you before you're even aware what is happening because that way you won't get it back and there's no room for negotiation.

I settled with running lxc containers. I just couldn't bring myself to be bothered about docker. Before I left my last job we started using k8s. I wouldn't use k8s unless in an enterprise environment. Even then I would consider the need. "But Mom, all the cool kids have k8s!"

It's not that. I used to own a VPS ISP. I know how the infrastructure stuff works.

It's that I don't care. I want to write code and deploy it to a new machine.

The time savings as a developer is enormous.

not the cloud means you write boring configuration files and call yourself a sysadmin

Also Rasmus (of PHP) wrote a series of posts https://toys.lerdorf.com/low-cost-vps-testing a while ago comparing VPS providers in the $10/mo range.

It was discussed on HN https://news.ycombinator.com/item?id=21725853

I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night.

In the morning, when I learned what had happened, my bill was over $3k.

I fixed the issue and emailed AWS asking for some relief, and they called me and let me know they were waving all the charges.

So, perhaps you too can beg for mercy?

Hypothetically, the contract could say Jeff Bezos will come to your house and personally kill you, but there's no consentual murder in most places

Now, Amazon would be entirely within their rights to cancel your account and refuse to do business with you after this, but they would not have the right to collect that money from you, or to keep that money had it already been charged to you.

Your only legal recourse under Title 18 Code 1030 is against the "violator". Amazon did not violate your computer systems and commit these offenses.

> Any person who suffers damage or loss by reason of a violation of this section may maintain a civil action against the violator to obtain compensatory damages and injunctive relief or other equitable relief.

On that basis, your contract stipulates who is responsible for fees associated with use of your AWS key by "any other third party".

> You are responsible for all applicable fees associated with use of the Services in connection with IAM, including fees incurred as a result of any User Credentials. You are responsible for maintaining the secrecy and security of the User Credentials (other than any key that we expressly permit you to use publicly). You are solely responsible, and we have no liability, for any activities that occur under the User Credentials, regardless of whether such activities are undertaken by you, your employees, agents, subcontractors or customers, or any other third party. You are responsible for the creation, distribution, and security (including enabling of access) of all User Credentials created under your AWS account, including credentials that you have used IAM to create or disclose to other parties.

The computer fraud in this case was not committed against you. It was committed against Amazon. Amazon grants you access to their services, the account does not belong to you. The damages here are not made against you, they are made against Amazon.

Just like in my example, the violator committed fraud against the "buyer" of the car. Neither Amazon or the "buyer" have recourse against you for the supposed owed property/bill, they have to extract damages from the violator. You are not responsible.

On your second point, I will repeat myself: it doesn't matter what the terms or contract say. Such agreements commonly hold terms that are in direct opposition to US law and have no legal basis. Their entire purpose is to dissuade you from pursuing your legal rights at a cost to the company.

This unfortunately isn't true. It also sounds like he created an app key from his root account that enabled anyone to literally impersonate him.

A typical use case is to create a user that has only the specific rights that are needed and generate an app key for that user. For example, I have a user that can only read S3 buckets. If it were to leak, the worst that would happen is I would leak some encrypted backup data.

In a situation like this, Lambda is almost pure profit. Their actual spend here was negligable.

They are almost certain to waive the fee, because they don't want the perception among developers that AWS is a time bomb.

My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours.

I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website.

Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k.

I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges.

I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access.

Lessons learned: 1. Customers may abuse things in ways you didn’t predict 2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills. 3. AWS can be reasonable about this but the ball is entirely in their court.

How popular is that vietnamese porn site!!

Getting to the important question here!

1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck for the benefit of learning that experience and getting your hands dirty with AWS.

2. AWS billing is actually really good for what it is. If you've ever run any non-trivial operational system (in the real world), you would know how hard it is to collate all expenses and get them tallied up. AWS collates all billing data with ~24h lag and you can slice and splice it to your heart's content. After all it's a complicated distributed system that they've managed to build that doesn't slow down your services or otherwise get in the way!

And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be.

> AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills.

If there were hard limits, there'd be no need to beg AWS support for leniency, which they can capriciously choose you don't deserve.

https://azure.microsoft.com/support/legal/offer-details/

Larger companies see $4k as a nothing; pay and move on. Household budgets, not so much.

What is true--not necessarily in order is:

- I suspect AWS in aggregate probably makes a fair bit of money on overages that a user eats but would have had a hard circuit in place if they could have, and

- Even reasonably designed hard circuit breakers (e.g. we cut off access to your stateful data unless you pay your bill but we won't delete it for 30 days) are still giving developers a potentially well-hidden foot-gun for a production environment that management might not actually want.

In real life when you hit your card's limit, your transactions get declined. Straight away.

I had this last month in a supermarket after my (personal) checking account didn't have enough money to cover my purchase, I'd completely forgotten to transfer money from my business account.

My bank wasn't prepared to let my account go overdrawn, not even by the equivalent of $20, which is absolutely their right.

Amazon, OTOH, benefits in lots of ways by not implementing this mechanism.

You put stuff in your cart.

You go through the register. You agree to "prevailing price".

No prices show up because they are "calculating".

You leave the store.

A day later, you're hit with a surprise bill that's 10-100x more than you though. A $100 transaction ends up being $1000-$10000.

There's no refunds.

The dispute procedure is to beg and hope they "let you ignore it".

Except for when it doesn't. I've got two primary current accounts, one with a "legacy" bank in the UK and one with a modern bank. The legacy bank is happy to let me go into an unplanned overdraft, and charge me for the privilege of doing so.

"Been there, done that". Not there any more!

At this moment, any dev with AWS keys has an unlimited month-per-month credit line that the company is on the hook for paying. And at best is the hope and prayer that the billing notifications aren't utter shit.

I've never run up a huge AWS bill accidentally, but I personally know 2 people who have, and neither was refunded, even after asking. In both cases we are talking $400-$800, enough to really hurt someone, but not bankrupt them.

The storage bill alone exceeded that at four startups I've worked with. When your job is to manage X PB of data with Y TB arriving daily, you're fairly constrained on the cost floor of your operations.

> You are making bad engineering decisions if you go through that in under a year.

Fine. You lack the imagination to conceive of a use case, or you lack the knowledge and skills to get the job doing it. Either way, this comment is rude and ignorant of the diversity of software challenges that exists in the real world.

The truth is that developers sometimes require expensive things to get their job done efficiently. The other truth is that we often vastly overestimate the SKU. And we also leave things running.

I wouldn't bet my bank account on that always holding true. If it's not in the terms and conditions that they'll refund you for accidental mistakes that lead to high billing, then you're gambling if you assume they will.

It could not have been easier to get AWS to remove the charge. A quick email to support with a brief explanation and it was immediately accepted. The hardest part was that they wanted a very specific request for how much I was asking to be refunded. So I had to go back and calculate my average costs per service and compare that to the charged costs. After that it was immediately refunded.

They aren't just handing these things out though. They made me read and acknowledge I had read their service agreements and basically swear that I know what happened and it won't happen again. Really painless process overall, all things considered.

> my usual bill is $200/month

How many req/second are you serving? What kind of things are happening?

It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month, including traffic and everything. I'm running a database on the same server, and 20% of the requests writes stuff both to disk and to the database, while the rest just reads from DB or disk.

The whole setup took around 5 hours to setup on one day, and have been running flawlessly from day one, we haven't had to migrate servers yet after ~6 months of production usage. Probably one day we're gonna have to upgrade the server to a $80/month one, but that's a one-time thing and our revenue easily covers that.

With a bit of refactoring and simplifying it tended to cut out a lot of issues. I suppose the issue is that a lot of people don’t really know what to look for or how to anticipate issues in complex infrastructure (which is totally fair, I only learned via trial by fire and have done some really stupid stuff).

I don’t think I ever encountered a case where the code was correct and bills were too high. Some AWS/GCP configurations would be pretty bad, but the code would also tend to be incredibly inefficient.

I always encourage people to respect people who are great at dev ops and to either hire them if they’re big enough or just consult them if they’re uncertain about things. Throwing a day of consulting rate at someone smarter than you is a great way to learn and it could easily save you money in the not-so-long term.

I’d say that because they’re crazy to rely on me for infrastructure, but I’d make things better if I was around already and they asked me to help out. But I’m no substitute for someone who actually knows what they’re doing.

I know I keep harping on about this, but [if you have a vague idea what you're doing] you can squeeze an awful lot out of a cheap VM.

I'm currently working on a project that we're deliberately prototyping using ultra-cheap VMs. If it takes off we know how to scale it up, if it doesn't the costs stay very very low.

but now, being a 1 person dev team, it is challenging in maintenance.

My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.

We, developers, never liked the reboots though, always wanted to find out root cause, so that we won't be paged again. So, I guess, we moved to the cloud. Now we don't get paged in the middle of night.

But yeah if it was my own company and I was on vacation, I rather take 30 seconds to reboot the server instead of worry about paying thousands of dollars.

As for lack of network coverage, you could do scheduled daily/hourly reboots while on vacation, if it makes sense for your service. Or if your an outage will cause massive disruption to your users, then perhaps hire a part-time sysadmin while on a vacation.

took about a day to rewrite and a week of 30 mins per day to figure out how to optimise my code for self healing.

knowing my bill is capped by number of VMs * memory, saves a lot of stress

For comparison, I've been running a paid Slack app (a few $K per year) that manages to run entirely within the free tier on Google Cloud Platform.

In any case the free tier will definitely allow you to handle enough traffic to figure out if you've got a viable business model, which I guess is the point.

These recursive requests were going at the rate 17k/second with 30s timeout each.

The benefits of the current approach is I don't need to manage any servers, and I have different environments for free. Also sizing is not an issue, I just tune AWS Lambda limits to be able to serve one single request.

I will need to invest some time to understand how big the instance should be (how much memory) because I struggle to measure how big it should be without going out of memory or CPU.

It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss or in unpredictable ways, but in most cases that would be preferable for these users as opposed to a couple orders of magnitude increase in billing costs.

But the cloud providers don't support hard billing limits because they like people fucking up and accidentally running up their bill. After all, it's probably only a small fraction of users that go through all the humiliating rigamarole of unwinding a provisioning mistake.

So yeah, good on Amazon for being so generous with the band aids, but maybe they should try a little harder at helping their users not shoot off their toes...

former AWS SDE here

I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime.

The effort to implement this is far from trivial, removes resources from implementing other features that the really large accounts are asking for, and adds complexity with direct availability risks. It's not at all surprising they don't implement this.

It's got 0 to do with why the feature doesnt exist though.

They wrote it off ~6hrs after we filed a support ticket about it.

- After crossing a certain threshold in scaling needs, Lambda costs more than regular EC2 on ELB

- Lambda cold-start times can be a deal-breaker when users first visit your website. If you contact AWS they will tell you to setup a simple cron job that keep lambdas "warm". But AWS provides no visibility in what's warm or cold, or which endpoints link to which lambdas.

- Dealing with Cloudwatch logs of various lambda runs (IMHO) is objectively a bad dev experience. Query insights is getting better, but is still a pain to work with.

- To reduce deployment and development times, you'll eventually want to deep-dive into lambda layers. Modern problems modern solutions.

- One lambda calling and awaiting another lambda is not a supported first-class use-case. There's no API that allows you to get the status of a lambda run. There's a hack around this where you use AWS Step-Functions. Modern problems modern solutions.

We're still on AWS full-stack "serverless" for our webserver and realtime stream processor. At the time I didn't know what I was getting my company into. I wish I just made a Flask webserver instead.

The technical details is outbound requests is given a role encoded in the user-agent, and then I can easily filter out incoming requests by user-agents [1].

[1] https://observablehq.com/@endpointservices/webcode-docs#opti... (see loop prevention flags)

After a surprise bill like this, I would re-evaluate what serverless is actually giving me.

> We compared the three-year total cost of ownership of a VPS, such as a DigitalOcean Droplet, against two equivalent leased or purchased bare metal servers. We estimated that the leased option costs about half as much compared to equal resources in the cloud, and owning the servers would cost less than a quarter of the pure cloud options.

[1] https://freebsdfoundation.org/our-work/journal/

[1] https://freebsdfoundation.org/wp-content/uploads/2022/06/Jou...

Everything on aws is a clusterfuck designed to suck money out of enterprise businesses

You're a small, three person group. Are you more worried about having a modest colocated server or a small VPS run out of resources because you suddenly became really, really popular, or are you more worried about who'll go without salary for a few weeks because you misconfigured something and got a bill for multiple thousands of dollars?

Amazon takes full advantage of the fact that you can get in to deep billing waters very quickly. They can't collect your billing info quickly enough? Bull - and I mean this emphatically - shit. If a company that runs some of the largest datacenters in the world can't communicate metadata in real time, they're either hopelessly inept, or they're failing on purpose to make more $.

I can respond up to 20k requests per second.

Scaling is for worldwide TOP 250 webs or webs designed according to the stupid contemporary fashions.

For example, it's totally understandable that the alarms can be specified per region, why shouldn't it be like this?

Also the global AWS billing $300 alert seems to have worked but you were asleep as far as I understand. If it was a call-out style alert, then you would've noticed in the middle of the night and could've stopped it.

The only thing I agree is frustrating is this: > CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).

Maybe you can argue that you weren't made aware of this but idk... keep us updated

Perhaps AWS should have "personal/developer" accounts that have this enabled by default and continually warn you about it, whereas "company/enterprise" don't have them.

Yeah, but I figure as long as Amazon doesn't immediately remove stored data, the damage of the footgun would be minimal. Speaking for myself, of course, I'd rather have a short outage than an unexpected thousand dollar overnight expense. It seems so trivial that it's unclear why AWS would not implement this feature. The only explanation that makes sense is that they want these surprise bills to occur.

Given how easily they reverse the bills, I suspect that they have a policy of doing it (perhaps a few times per account, something to prevent abuse) because they really don't want to trigger the above scenario.