No mention of how AWS/S3 approximates the size of a file to save CPU cycles. It used to drive me up a wall seeing S3 show a file size as slightly different than what it was.
If I recall correctly, S3 uses 1000 rather than 1024 to convert bytes to KB and MB and GB. This saves CPU cycles but results in “rounding errors” with the file’s reported size.
It’s discussed here, although they’re talking about it being “CLI vs console” which may be true?
This almost certainly has nothing to do with “saving CPU cycles” and is most likely just that whoever created the Cloudwatch console used the same rounding that is used for all other metrics in CW, rather than the proper calculation for disk size, and it was a small enough issue that it was never caught until it was too late to change it because changing it would disrupt the customers that have gotten used to it.
Yeah we have long standing confusion that for historical reasons KB and MB often means 2^10 bytes units so now when you see KB you really don't know what it means. Therefore I am a staunch supporter of unambiguous KiB and MiB.
I think I explained this poorly, and I appear to be mistaken about it being to save CPU cycles (though an entity such as AWS would absolutely be about saving minuscule CPU cycles that add up at scale).
I spent a lot of time researching this issue when I came across it years ago- I just remember that local file size and S3 file size was not matching up with anything and the takeaway was that S3 was calculating file size differently from Windows/Linux/macOS.
AWS uses base 2 (binary) for calculating file size, meaning 1MB is treated as 1,048,576 (2^20) bytes rather than 1,000,000 bytes (10^6).
And as you've said, one is MB while the other is technically MiB.
If I recall correctly, S3 uses 1000 rather than 1024 to convert bytes to KB and MB and GB. This saves CPU cycles but results in “rounding errors” with the file’s reported size.
It’s discussed here, although they’re talking about it being “CLI vs console” which may be true?
https://stackoverflow.com/questions/57201659/s3-bucket-size-...