Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

In the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?


I worked with a couple of the commercial alternatives to aspera and aspera as well and I used UDT, Tsunami, GridFTP, syncthing[1] as a poor mans alternative to Aspera.

For real transfer with big (> 500TB) Aspera will deliver what it says over WAN (over the Atlantic).

If you have better connections and not so much data syncthing will probably work if you can have someone manually take care of all exceptions.[2]

If you know your data and you can build quite a lot of stuff yourself you can get almost the same speed as Aspera with UDT.

There is also a go implementation of UDTs used by kcptun[3] but I haven't tried that.

1 https://github.com/syncthing/syncthing

2 I don't intend to disrespect syncthing here, but when your dealing with TB scale 24/7 things are never really up all the time, the network is unreliable, the sender host filesystem is corrupt and the receiver host filesystem is full or broken or ...

3 https://github.com/xtaci/kcptun


Has anyone had good experiences with KCP? Syncthing removed their implementation [1] because it didn't live up to expectations.

1 https://github.com/syncthing/syncthing/issues/4737


Yes, I got a similar list in response to my link on Twitter https://twitter.com/fanf/status/1210858696558944256

But it looks like GridFTP and Globus are basically dead https://opensciencegrid.org/technology/policy/gridftp-gsi-mi...

Tsunami sounds good but maybe needs some updating? I haven’t looked closely...


Globus is definitely not dead[1], it has an active community with strong uptake in the Materials Science world (where I work), among others. The Globus folks have a repo for many popular Linux distros, it's reasonably straightforward to get started. (I'm a user of it, not a developer or necessarily an advocate.)

It's not free for "managed" (Enterprise) applications, though, which is what the OP seems to be looking for, and I'm not sure it's the best choice for high-speed.

Something that I think is free is the CERN FTS [2], which can use a GridFTP back-end, so possibly you can roll your own high-speed big-data infrastructure that way.

[1] https://www.globus.org [2] https://fts.web.cern.ch/


I concur that Globus is definitely not dead! We're transferring 10s to 100s of TB per month with it. Globus is continuing to maintain GridFTP for this purpose, as part of Globus Connect Server and Globus Connect Personal, and it will happily saturate a multi-Gbps link.

And personally, I'd like to see more people getting a Globus subscription. It's cost-effective when compared to tools like Aspera, and helps fund the development of Globus software and features.


Globus isn't dead at all globus.org, it just turned into something that solved an actual problem.




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: