Ex Aspera Dev here. I did the encryption and early parallel work.
There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery.
The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!", and did. So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises.
If you think this wouldn't be able to go a thousand times faster than TCP, you have never tried moving a file to China or India over TCP. :-) (Customers used to report 5% drop rates.) Drops and high RTT are devastating to traditional TCP throughput on high-packet-rate routes; read about "slow-start" sometime, and do the math. Problem is that for untrusted peers, drops are the only trustworthy signal of congestion. Recent improvements where routers tag packets to say "I was really, really tempted to drop this!" help some.
Torrents get the out-of-order delivery and the lower sensitivity to drops, but its blocks are too big.. Others commented that opening lots of connections gets around some TCP bottlenecks, but that helps much only when the drop rate isn't too high (i.e. not to India or China).
I'm curious how much the observed speed of Aspera on the Internet might rely on "nicer" protocols (TCP) backing off as Aspera causes congestion^H^H maximises link utilisation.
Do you think the protocol might cause problems if it were more widely used?
Interestingly (at least, I find it interesting) fasp can happily use up all the bandwidth the various TCP streams aren't using without affecting TCP rates at all. It also backs off and shares bandwidth with other instances of itself or other well-behaved protocols. With some coordination between them, multiple senders can share the available bandwidth in any chosen proportion.
So, you can set up where A always gets 70% when it is sending at all, and B, C, and D share whatever is left when they have anything to send, but all without ever slowing whatever TCP traffic is running. An administrator can (literally) drag up and down the rates of ongoing transfers according to organizational priorities.
That sort of featurism is part of why Aspera can command crazy prices.
It sounds like people are getting good results with fdt. It leads me to wonder if fdt has any of this good-network-citizen capability. A lot of people using fdt and making the net worse for everybody else would be an unfortunate outcome. Aspera was pretty careful about it.
(I keep trying to write ftd, but that is the florist service.)
Rate control sounds nearly identical to TCP Vegas then, no?
But out of order delivery and (I'm guessing) out of order retransmits appear to be unique to fasp. What did you mean about these being unique "until recently"? [edit - nevermind, saw below you were referring to sack]
PS. I remember looking at fasp ~10 years ago and it looked like a fantastic tech. Someone's taking things that many people only talked about and putting them into a product that really worked well. Cutting edge stuff. A job like - a dream for many :)
No, quite different from Vegas, if the graphs are to be believed. What I would see on real trans-oceanic links was a sub-second ramp up to a flat, stable rate, and stepwise rate changes during the session as other flows started and stopped, with no visible oscillation.
Fasp does not rely on RTT ACK timing, which is very noisy, but only outgoing delays.
The routers are applying their own algorithms to simulate sane queue timing, but they are doing all kinds of crazy shit under the hood to get better link performance. The queues, IOW, are a fiction provided to the endpoints to simplify their job and generate understandable, thus tunable full-route dynamics.
But the ACKs get completely different treatment, because traditionally nothing cared about ACK timing.
So, "onset of queuing", e.g., is pretty meaningless in practice, and fasp makes no attempt to detect it.
I was not familiar with it. It looks like you could build a good-enough file transfer system on top of UDT, but probably not with the rate control protocols they provide.
One problem with UDP protocols over the open internet is that you cannot trust UDP packet contents, which could come from anywhere.
Step one: send a file that's cryptographically signed.
Step two: the client solves the problem of "which of these duplicate and conflicting packets do I need to assemble together to make a message that is appropriately signed?"
Didn't use to be. VIA/centaur was the first to add AES instructions, later picked up by Intel and AMD. Its not free, but not expensive.
A brilliant woman from, where, Slovenia? figured a way to use SSE instructions to encrypt just as fast as the built-in instructions, I think 2 cycles per byte.
I guess DJB's ciphers run much the same way, nowadays.
not true, although i'm sure there are one-off instances where there's no encryption. but in general there's lots of encryption on each of the different levels of secure government networks.
Right, they didn't want us encrypting what they were going to encrypt in their routers anyway. "Trusted" is a technical term.
So they started using Aspera to send video from the UAVs over Iraq etc. to the Pentagon. Before that, they were literally flying boxes of videotapes (or CDs? Tapes seem hard to believe, except this was govt.) to a warehouse, where analysts could check them out, like from your neighborhood library.
Probably now it's all realtime. But they have to stream it, somehow, so it is likely still going via Aspera to Nevada or wherever they work the remote pilots.
At the time (mid-late 2000s) the video output from the UAVs was sent in the clear, in analog format, and recorded in analog format. So they had to start digitizing it in the field to transfer as files.
Jihadists could tune in an ordinary scanner, and watch the video coming from them. Embarrassing.
This is something bittorrent could get better at. It would be cool to update the bittorent spec to use eg. a resnet to predict endpoint and path saturation from moment to moment.
It's kind of amazing really that the near future will probably be about removing massive amounts of code & design from all sorts of incredibly carefully engineered systems like the linux kernel in favor of a pile of linear algebra that can just figure things out. resistance is futile.
If you think the best answer to predicting things is to just throw a neural net at it, you should probably learn more control theory. :)
(Less flippantly - neural nets are mainly for the case where you can't effectively do manual feature engineering because your inputs are too varied and complex, you don't particularly care about the mathematical properties or guarantees of the solution, and where you don't even really know what the solution should look like [so are forced to randomly initialize a huge nonlinear system and optimize it until useful behaviors pop out]. Something like predicting path saturation would be far better suited to traditional approaches.)
I was thinking in terms of relative human effort. Adding a neural network into an open source project like bittorrent is going to be very fast to integrate and will probably offer performance that meets the satisficing criteria of being instantly better than the currently offered level of performance.
Edit: By the way, I accepted your initial opinion about traditional approaches being better at face value but that was like ~45 minutes ago. After thinking about it a bit more I realized that there is alot of interesting network data that could be fed into such a NN, moving it (ding!) back into the nonlinear/complex system column. Suddenly my approach is looking competitive to also being the optimal one (excluding hybrids).
I believe it would be an interesting exercise if you let the system decide everything, like how often the data is sampled, what information is shared and with whom, etc. Information it might share could be time of day, location, all sorts of routing data, data hashes, # of connections and host throughput rates, and so on. I just did a very quick search on arxiv about leveraging ML algorithms in software defined networking. My guess is that the starting state assumptions that are most important are whether it's a centralized (homogenous) or distributed (heterogenous) network. A fun problem.
Do you have any anecdotal experience with how this sort of approach affects very short, bursty stop-and-start transmissions (notwithstanding the fact that you obviously can't go faster than link latency)?
Doesn't TCP have an ability to re-send dropped packets, now, without sending everything after them? I thought they called it selective re-send, or selective ack, or something. Maybe the receiver could send an ACK with a bitmap of which packets it got, and the sender would fill in the holes?
I believe it's Selective ACK aka SACK. It allows TCP to ACK ranges of packets so the sender can see that it only needs to resend an earlier block.
I've seen rather good speeds over some bad links from my bittorrent server using large window sizes (up to 9), BBR flow control, ECN fully enabled, and SACK.
Yes it has had selective acknowledgments(SACK) for quite a long time(RFC 2018.) I thought the OP was referring to something perhaps more recent since I believe SACK might predate Aspera.
Most of our customers didn't have SACK-capable network stacks. It used to take a very long time for IETF specs to find their way into deployed systems.
> The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!" (and did). So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises.
Would you be willing to talk more about this, for someone not as intimately-familiar with the details?
I've reported most of what I know. I was involved in the data layer. Flow control was done by the PhDs.
But, imagine the route as a series of packet queues. Packet is sent, arrives at the first router, spends time in the queue, gets sent, put on the next queue, waits, goes, until it's delivered. If the total time taken, compared to the last packet's, is more, that means the queues are growing. If the total time is less, the queues are shrinking. You adjust your rate to stabilize them. Other flows are being added and removed all the time, so the optimal rate varies in steps over the course of the session.
This involves control theory, Kalman filters, PID, the whole business.
As a general question, I'm curious how a tactile learner might be able to get a decent grasp of pretty much everything you just described, in a practical setting, without breaking the bank.
:)
How good is GNSS at simulating latency, buffer saturation, etc on a simulated network and how much complexity would I need to contemplate factoring in to build an accurate model of the average organically evolved, ad-hoc, multi-WAN-spanning sprawling corporate network?
I can't think of a more concrete, instructive starting point for someone interested in figuring out an alternative/reimplementation.
It really is a matter of learning the hard way the phenomena that show up on real networks in the wild. Aspera spent a decade incorporating responses to all these lessons into their code.
You can get 80% there quickly, 90% there in two years, but the rest is just hard. A versatile simulation environment feeding the algorithm with synthetic time events according to a configured schedule is probably the best way to ramp up quickly, but I don't know anything about GNSS.
One tricky phenomenon is bimodal and even N-modal delay samples from traffic split across different routes.
Around 2014, Aspera replaced most of the code wrapped around the algorithms to become much more nimble about applying them in more circumstances.
I see. I get the impression Aspera is built on a really simple solution, "pressure-adjusted" over a very long time to account for the fact that the real world isn't a vacuum :D
Hearing I can get to 80% quickly is encouraging from a MVP viability point of view, thanks. (And also from a fail-fast perspective; I've now gotten to wondering how much additional performance might be eked out of start-stop style traffic, and it's good to know it won't take long to find out whether the pursuit is worthless or not. Yay.)
I read the second half of that second paragraph as describing a network model simulation that "compiles" a particular routing graph/topology into a set-in-stone sequence of packet events that you then later analyze...? (I'm interpreting "the algorithm" is my target application code, and "synthetic time events" and "configured schedule" as hints at non-real-time pregeneration. This may be incorrect.)
Eek, the delay distributions you describe almost sound like they might be NP-complete to solve.
The idea of fast simulation is to feed just your sending-rate update algorithm with fake time samples, with nothing like actual packets anywhere, so you can get very quick evaluation of your algorithm's numerical behavior under a wide variety of conditions.
Routes tend to start monomodal and split and rejoin at specific times, so it is not usually as bad as you imagine. Also, the number of routes does not get large.
(Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high. In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 900 mbits/s without needing any TCP features). This is also helps high latency scenarios, as you don't need test/back off how much data to send...you just send as much Raptor encoded data as you want down the pipe, and as long as you send enough recovery packets with it, you will be able to reconstruct the original flawlessly).
You don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs).
Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.
Yes, of course--even a low rate of packet loss where the bandwidth delay product is high will cause you to end up with throughout dropping to critically terrible levels. Tweaking TCP can help, but a high performance RaptorQ system can work at packet loss levels that TCP could never handle.
No, the overhead is fairly minimal. A 5% packet loss (which is very bad) on a gigabit link where the sender is thousands of miles away might have an effective throughput of 930 mbits per second with RaptorQ. With regular TCP you would get only a few megabits/s, as it would back down until the loss rate was nearly zero.
Second this. When we needed to deal with somewhat similar issues (quickly transfer multi-gigabyte archives over massively unreliable link) , we ended up with FDT ; it can be easily scripted , and transfers data between two hosts (without using a network of torrent peers) at that hosts' link speed. Used it to transfer builds between US and Russia, and Russia and China .
The 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to 00:09. First of all, this is 15x faster, not 200.
I tried downloading the file using axel -n 32, and it took 1:28, which is only 9x slower. Curious though, when it started out with all threads running it was transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by the end with only a few threads left running it was only doing 5625.8KB/s.
It looks like the performance varies a lot, possibly due to multiple interfaces or wan links being used.. or issues with their server.
Using axel -n 64 over http I was able to fetch it in 50s, which is only 5x slower, and most of that time was spent from 98 to 100% as the last few connections finally finished.
A smarter client that more aggressively re-fetched chunks being downloaded over slow connections would likely match the same performance.
not to take away from the posts plea. Apart from bit vs bytes, it should be noted the remotes are also on different machines with potentially varying network throughput
Why not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer.
If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the data after transfer.
Aspera only uses multiple UDP streams if the line rate is too fast for one sending process to keep up. Very often the actual bottleneck is the receiving storage backend.
It absolutely does not measure packet loss at all. The only response to a dropped packet is to request a re-send.
The protocol is simple but depends on very smart sending rate control. Much of the complexity of the current version is to perform well when the receiver has a slow disk, or the link goes through a satellite.
Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast.
Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.
You're not wrong, but the opening statement "Zip is now effectively obsolete." is sort of true but not really in a lot of contexts. It isn't something I'd say with conviction.
From your point of view was the implied statement something akin to: "In the context of a high-throughput network transfer program choosing DEFLATE over an algorithm more well optimized for computational weight wouldn't be a good idea." ?
I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator.
It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs Altavista, Git vs Subversion vs RCS. Need I go on?
Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known.
The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.
OK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.
I thought ZIP files only supported a bunch of old algorithms like deflate. Maybe you can make an exotic zip file with zstd but doubt there are any tools that will understand it.
I benchmarked Aspera against alternatives for getting data from on-prem storage into the cloud (google cloud storage) a year or two ago on a 10Gbps link.
Aspera was faster on a single stream, but if you have a lot of files to move around (you usually do) you can just multiplex a bunch of tcp streams to get the same throughput.
So like in the example on their page, if you have 20 wget's hitting their ftp and there are no other bottlenecks, the throughput will be similar to using Aspera - and you have just a greater variety of free tooling for tcp based those protocols...
Sending on lots of TCP streams can get you to 80% line utilization if the drop rate is good and low. That's often good enough. On some lines, with the "wrong" number of TCP connections, the rate will oscillate, and you will be lucky to get 50%. Tuning that is a black art some people enjoy.
Yeah, I was probably working in an environment with pretty low packet loss.
Looking at the problem holistically - if you're going to spend money on Aspera to fix situations with high packet loss, maybe you're better off spending money on a higher quality connection and using tcp.
Yes, that works when latency is low. On trans-oceanic links you often have to take what you can get, drops and all, and physics has something to say about latency.
There is no reason fasp couldn't be reverse-engineered and deployed for free. But it involves real engineering. Once you did, you would have something valuable, and might be inclined to milk it. Aspera has lots of lock-in now for ancillary features of the product -- monitoring, encryption in flight, encryption at rest, bandwidth use scheduling, yada yada; but in the early days, competitors (Signiant, Riverbed) had all that, but not the flow control sauce, so failed on difficult routes. "Enterprises" like not to care about the difficulty of routes, and it's amazing how many consider the license cheap.
Even if you have only the one large file, can't you segment it? That's what I do for my consumer level use case anyway. Using a nonsegmented ftp client to download a file I get like 2MB/s, and when its segmented I max out my download rate at 20MB/s.
Honestly speaking, a good way to solve this is not trivial. You would need to:
1. Measure the existing performance on specific data sets.
2. Understand how many of the bottlenecks come from the network latency vs. I/O limits vs. CPU bottlenecks (if using compression).
3. See if any domain-specific compression is needed.
4. Document the typical use cases. Understand why the current solution sucks (e.g. requires redundant user actions). Write down user interaction scenarios. Design the UI to be as efficient as possible for those scenarios.
This is a non-trivial amount of work that would require a lot of back-and-forth interaction and on-the-go requirement changes and I don't think it's entirely honest to ask someone to do this work for free in the name of cancer research (after all, you are not donating most of your paycheck to charities, are you?). If the existing solution by IBM sucks, how about making a Request For Proposal [0] and seeing if smaller software vendors could offer something better given that you are actually willing to pay for the work?
P.S. A student/hobbyist can probably whip out some sort of a parallel TCP-like thing with a large window for free, but you would get the same performance by just cranking up the TCP window size via sysctl and using multiple HTTP threads (htcat was suggested earlier in the comments).
Way back when I had a problem on my network that was capping the FTP/SFTP speeds to about a fifth of my download speed. Torrents worked fine. To fix my problem I have cobbled up an extremely hacky solution which consisted of opening several listening sockets, ssh-ing to the remote machine and net-catting the file by chunks in several processes in parallel. It actually worked. The code is here https://github.com/jlegeny/blazing-speed more for archiving purposes than to be used by anybody for anything.
FTR, they say: "As you can see, the ascp client performs over 200x faster than FTP, and can download the whole file in 9 seconds. It’s pretty magical." But the example shows MB/s vs Mb/s. It's not over 200x times faster, but about 17 times faster.
Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data?
Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections?
Also, is there any room for an rsync type approach, sending compressed deltas rather than naively sending huge files that may be redundant?
Not to say disintermediating a BigCO expensive patented vendor-locked-in MLPOS (Market Leading Piece Of Shit) doesn't sound exciting -- it does. I'm just curious and a bit skeptical that it is always necessary to mass-copy all this data over and over.
Aspera is the standard in broadcast video for transferring video files. Typically you wouldn't want just a small piece, you would want the entire hour-long, 3 terabyte video.
Also, you would still need to get the data into the cloud in the first place.
The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.
Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
Sure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual techniques for general compression, but it's going to limit how much you can compress without information loss. In addition, the sequencer assigns an unique ID to each sequence read. It's often ok to discard this information, but occasionally you need it in order to trace a specific problematic sequence through the entire analysis pipeline. So if you want lossless compression, you have to compress these as well. They should be rather compressible, since they are typically a prefix global to the file followed by a number that increments by 1 for each sequence. But for a general compressor you can't just automatically assume that, because according to the (lack of a) file format spec, each ID could be anything as long as they're unique within the file. So if you want guaranteed lossless compression of any FASTQ file, your compressor needs to be sufficiently general.
Beyond that, compression of sequences by encoding only the differences from a specific reference sequence somewhat ties you down to using a specific reference, since that reference sequence is required for decompression. This is inconvenient because the reference sequence for a species is updated over time, and if you want to use the new reference, you'll need to decompress with the old reference and then re-compress with the new one. Do you want to do that for all your data, every time the reference is updated? Reference-based compression also means the compressed files are no longer self-contained, which may not be acceptable for certain use cases.
None of these issues are fundamentally impossible to address, but the point is that it's not nearly as simple as it sounds, and anything fancy you do to try and compress a sequence file will generally impose some additional requirements on what the receiver needs to do in order to read the file. And many receivers are not very technically inclined researchers whose plates are already full of other things they need to be doing besides figuring out how to decompress your new unfamiliar sequence compression format,
I feel the author’s pain - Aspera is a clunky 90s styled Ruby on Rails app that is about the most cloud unfriendly piece of software I’ve encountered, I’d love an alternative.
I honestly don’t care about about their proprietary UDP protocol, it’s nothing special, just another way to copy bits onto a wire. Dime a dozen.
The true value of Aspera is they provide an integrated browser plug-in that lets the technically challenged reliably upload large files. If the transfer is interrupted or either side changes addresses it deals with it gracefully. It’s also a bridge to AWS S3.
I spent some time looking for a replacement and while there are numerous download managers that facilitate people downloading large files, I couldn’t find any upload managers with the same level of integration and polish.
About the closet thing I could find is Cyberduck, but it’s not integrated with the web browser, not as easy for technically challenged people to use, and there is no support (community support but seems really hit and miss). However it does make good use of Amazon’s multiple upload api and will happily fill whatever wire it’s connected to.
Torrent software has largely the same pros and cons Cyberduck does.
I have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?
Used a few of their products at a previous company. That's the gist of the protocol. It's UDP-based and the software has intelligence to use bandwidth as efficiently as possible between links. We had a few servers sending to many endpoints concurrently, and I'd regularly see the systems cap out their 10gbit connections.
License costs are pretty brutal and it does need decent amount CPU to make the most of it.
Are you able to recommend a resource for how to use bandwidth most efficiently like this? I'm interested in learning more, and would appreciate it. Also, might using raw sockets not offer better performance and lower CPU usage?
It does what it says and it’s expensive. There’s a couple of commercial packages like this - two that come to mind are vcinity and Signiant. A solution can be made but every time I look at the market you find very few customers with a real need to move terabytes that can’t solve their problem with a snowball-style solution. Those that do usually also have the economics to justify a commercial package.
Don't know about vcinity, but Signiant was always a marketing-heavy, engineering-light organization. Their focus was on user convenience features, and they almost ignored actual transit engineering.
I know about a couple of these. Some use very specific compression based on external knowledge—like that this is all office xml documents, with freedom to study the set of the last year’s documents. Some work by eliciting pessimal behavior from TCP flows sharing the same link—they’re really bad neighbors!
Some use a family of different approaches to cope with different performance domains. Some use intermediate relays to cut the bandwidth-deat product by reducing delay.
But the big key is this: standard tools aren’t even trying to be good at this. They’re aimed at making an Internet work, not at optimizing and particular flow.
It works, maybe not thousands of times faster, but it’s a fast protocol.
Basically it chunks up data and pushes it out as quickly as possible with UDP then retransmits whatever is needed. It works great in enterprise networks where you have high packet latency due to shitty firewalls and inspection appliances.
I haven’t used it in awhile, but I think they also have a proxy like service that lets you do managed file transfer directly from your datacenter network instead. It’s possible that I was using a complimentary product to do that.
Aspera user here (Well I used to, before it was bought by IBM)
Its only faster over large high latency links. Instead of using a single TCP stream to push data (with that exponential roll off) it has a TCP session to do accounting, and a bunch of UDP ports to stream the data.
for links around 16ms latency, its not that much faster.
For a 10 gig link with a latency of 100ms, it runs pretty much at wire speed. (assuming you have a machine that can handle it.)
Orders of magnitude improvement over TCP is easy when the bandwidth-delay product and loss rate keep TCP stuck at 0.1% of line rate. It says more about how bad TCP is on those lines than about how good the alternative is.
But the alternative is good, too. Worth the price? Depends on non-technical factors.
If they are like Riot and have their own fiber, it is not unlikely. There are tons of boring optimizations open source software also won't bother with without significant sponsorship (some could be specific to a particular file format). If you look at most network (and also video) protocols, they are accumulations of decades of improvements rather than neural network style overnight breakthroughs.
MIT open courseware has a course on Performance Engineering which details some of the types of low level optimizations that can be done. It requires a lot of measuring and problem specific solutions.
I’ve helped with that course in the past. It’s awesome. It’s completely unrelated to WAN networking. I found the IETF groups working on these problems more helpful.
bbcp doesn't work so well for some reason. Where I used to work we had a senior dev that claimed that he could just repurpose bbcp for this. Then he gave up and wrote something custom in c++. Then he rewrote it in go, but used (among other things) non-multithreadsafe primitives, then when asked to make it encrypted, he used epoll with tls (which is apparently not a thing in go? I don't know). I told him he should just use DTLS or hell even a one-time-pad encrypted UDP stream with backpressure management, but he didn't listen to me. Then he rewrote it in C++ again and then went back to the go version. When I left the company, it still wasn't working.
Also, this senior dev never wrote unit tests.
On the other hand, maybe bbcp will work, and the senior dev just didn't know what he was doing. There was another senior dev who, seeing what was coming down the pike, left his job, and on his way out he was like, "yeah you can do it with bbcp, just you gotta tweak your tcp congestion rules on all the hops (which we could do, but is probably not an option for OP)"
The other senior dev was pretty close. We support bbcp for transfers to/from some of our environments, but we don't do the tuning. Instead, we set things up to be as accessible as possible (jumbo frames enabled, minimum # of hops from the DTN to the border), and leave it to the user to set all the bbcp options.
I worked with a couple of the commercial alternatives to aspera and aspera as well and I used UDT, Tsunami, GridFTP, syncthing[1] as a poor mans alternative to Aspera.
For real transfer with big (> 500TB) Aspera will deliver what it says over WAN (over the Atlantic).
If you have better connections and not so much data syncthing will probably work if you can have someone manually take care of all exceptions.[2]
If you know your data and you can build quite a lot of stuff yourself you can get almost the same speed as Aspera with UDT.
There is also a go implementation of UDTs used by kcptun[3] but I haven't tried that.
2 I don't intend to disrespect syncthing here, but when your dealing with TB scale 24/7 things are never really up all the time, the network is unreliable, the sender host filesystem is corrupt and the receiver host filesystem is full or broken or ...
Globus is definitely not dead[1], it has an active community with strong uptake in the Materials Science world (where I work), among others. The Globus folks have a repo for many popular Linux distros, it's reasonably straightforward to get started. (I'm a user of it, not a developer or necessarily an advocate.)
It's not free for "managed" (Enterprise) applications, though, which is what the OP seems to be looking for, and I'm not sure it's the best choice for high-speed.
Something that I think is free is the CERN FTS [2], which can use a GridFTP back-end, so possibly you can roll your own high-speed big-data infrastructure that way.
I concur that Globus is definitely not dead! We're transferring 10s to 100s of TB per month with it. Globus is continuing to maintain GridFTP for this purpose, as part of Globus Connect Server and Globus Connect Personal, and it will happily saturate a multi-Gbps link.
And personally, I'd like to see more people getting a Globus subscription. It's cost-effective when compared to tools like Aspera, and helps fund the development of Globus software and features.
> Aspera is owned by IBM - does their client spy on what else is running on my system and report it back to their headquarters as business intelligence? If asked, would they share this information with the government? Without the source code, it’s impossible to tell!
I mean, with some reverse engineering, mentioned later in the post, it’s really not.
My group is about to start using Aspera as well.. I too wish there was a good alternative I could suggest to my bosses. And I'm still skeptical of their claims... but I haven't gotten to test it over an actual bad link yet (over a good link it does nothing, which a simple test will prove).
It's being run out of UChicago, and I know they have at least one large company using them. Feel free to reach out to the Globus team, or shoot me an email!
Aspera never did in-flight compression when I was there. In-flight compression got even less interesting around 2010 when the long fiber links got faster than customers' (shared) storage systems. Users have good reasons to compress at rest, before starting a transfer.
Certain labs were sending neutrinos right through the earth, gaining a measurable reduction in transit time vs great-circle overland signal propagation. (At first they thought they were gaining even more, by superluminal transit, but that turned out to illusory.)
They did not put any data in the neutrino channel, to my knowledge, but in principle they could have.
You might think you would need quite a deep chord to beat microwave propagation over the surface, because the dielectric coefficient of rock reduces the speed of light there. But that wouldn't affect neutrinos, which definitely go much faster through any kind of rock than light ever goes down a fiber. True fact.
Ah, I see. So the labs you refer to were trying to make this work? I'm curious what sort of performance level they achieved, even if eg just multiples of single or tens of bytes/sec.
Bear in mind that you have to put the signal in and get it out of the channel as well as transmit it. I highly doubt that neutrino signaling kit is as optimized for doing this as photonics.
There's a meta answer about the general approach to such problems: Ask what other research areas must have tackled it, and then find out what they do. The obvious answer in this case is HEP, which would lead to some of the tools mentioned. (There's also https://www.psc.edu/index.php/hpn-ssh but I don't know how it compares.)
It's unfortunate how many researchers don't do such research...
Beyond more profound changes to the underlying transport protocol, bandwidth acceleration can be provide through a combination with fast compression.
A good setup here would be to pipe data to `tar`, then `zstd`, then `ssh`/`scp`,
and do the reverse on the receiver side.
From an end-user standpoint, all this could be abstracted away if one could get `wget` (named in the article) to support zstd compression natively, which is possible since zstd is also a web compression standard.
Such compression support is, by the way, available in `wget2`.
ZMQ is too unreliable at transferring data which is a fraction of the stated use-case. Streaming realtime data, ZMQ had problems dropping data unpredictably under load testing, so ZMQ never made it out of testing for us. The protocol is too naive.
There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery.
The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!", and did. So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises.
If you think this wouldn't be able to go a thousand times faster than TCP, you have never tried moving a file to China or India over TCP. :-) (Customers used to report 5% drop rates.) Drops and high RTT are devastating to traditional TCP throughput on high-packet-rate routes; read about "slow-start" sometime, and do the math. Problem is that for untrusted peers, drops are the only trustworthy signal of congestion. Recent improvements where routers tag packets to say "I was really, really tempted to drop this!" help some.
Torrents get the out-of-order delivery and the lower sensitivity to drops, but its blocks are too big.. Others commented that opening lots of connections gets around some TCP bottlenecks, but that helps much only when the drop rate isn't too high (i.e. not to India or China).