From my experience (not with amazon) these strange chunk sizes come from non-blocking IO. When a source gets some data and triggers select/poll/epoll/whatever, the callback (or equivalent) immediately writes it out as a chunk.
This works even better in HTTP/2 or HTTP/3 / QUIC. A Go server reading from a lot of microservices can produce pretty weird output on HTTP/2 because now not only is it in odd sizes determined by network timing, it doesn't even need to be in order.
This works even better in HTTP/2 or HTTP/3 / QUIC. A Go server reading from a lot of microservices can produce pretty weird output on HTTP/2 because now not only is it in odd sizes determined by network timing, it doesn't even need to be in order.