Why not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer.
If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the data after transfer.
Aspera only uses multiple UDP streams if the line rate is too fast for one sending process to keep up. Very often the actual bottleneck is the receiving storage backend.
It absolutely does not measure packet loss at all. The only response to a dropped packet is to request a re-send.
The protocol is simple but depends on very smart sending rate control. Much of the complexity of the current version is to perform well when the receiver has a slow disk, or the link goes through a satellite.
Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast.
Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.
You're not wrong, but the opening statement "Zip is now effectively obsolete." is sort of true but not really in a lot of contexts. It isn't something I'd say with conviction.
From your point of view was the implied statement something akin to: "In the context of a high-throughput network transfer program choosing DEFLATE over an algorithm more well optimized for computational weight wouldn't be a good idea." ?
I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator.
It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs Altavista, Git vs Subversion vs RCS. Need I go on?
Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known.
The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.
OK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.
I thought ZIP files only supported a bunch of old algorithms like deflate. Maybe you can make an exotic zip file with zstd but doubt there are any tools that will understand it.
If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the data after transfer.