I think every developer has an AWS billing horror story.
My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours.
I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website.
Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k.
I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges.
I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access.
Lessons learned:
1. Customers may abuse things in ways you didn’t predict
2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills.
3. AWS can be reasonable about this but the ball is entirely in their court.
Worth noting that if your distribution is set to use every region, Asia-Pacific CF pricing is actually more expensive than raw S3. $0.12 vs 0.09 (for S3) or 0.085 (for North America CF). It's easy to accidentally increase your costs by 33%, since you'd only encounter this with S3 if you put the bucket in an asian region, versus CF where distributions are more hand-wavey about locations.
My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours.
I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website.
Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k.
I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges.
I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access.
Lessons learned: 1. Customers may abuse things in ways you didn’t predict 2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills. 3. AWS can be reasonable about this but the ball is entirely in their court.