Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: SwimOS Rust – A framework for real-time streaming data applications (github.com/swimos)
71 points by swimos 6 months ago | hide | past | favorite | 22 comments



The README tells me mechanically what swim does but even with the example it’s quite abstract and I don’t know why I would choose to build with swim, what it’s similar to, or what it’s especially good at.

I think with frameworks you should make the value proposition very clear otherwise it’s a tall ask for someone to adopt a way of building. Versus a library which is easy to drop into existing software to solve a specific, scoped task.

I guess a bunch of what I’m looking for is discussed in the main repo of your project, but even there it’s a bit heavy on the “what” and too light on the “why” for me. This is a deep framework. I work on a realtime reactive app and am currently building stateful services on Kafka + Typescript + SQLite, so I think I’m squarely in your target customer demographic.

It’s not until I found the website that the value proposition is clearly described. I encourage you to copy-paste the headline of your website to all the components of your framework, so it’s easier for a casually interested person on HN to grok what it’s all about.

EDIT: there’s like a 1% chance I actually do convert, since I realize I could run my existing Typescript business logic in a Rust agent via Deno… perhaps the scattered documentation hunt on a Sunday is good marketing strategy after all.


I had a similar reaction, and assumed I'm not the target audience. (Including skimming the examples folder and not being sure where I would use one of them) I have a hard time understanding the purpose of these abstract libraries; I assume people who have had the problems they solve recognize their utility.

As a reference point for rust libs I've worked with recently: I see things like EGUI: This clearly is a library that allows you to add a GUI to programs. Or Bio: This lets you read and write FASTA format DNA etc sequences, find matches in sequences etc. Bincode: Provides a "derive" that allows you to easily serialize data to and from binary formats. Or the various embedded infrastructure libs that provide high-level APIs for performing hardware options. (I/O, ADCs, send a packet over USB or a radio etc) By contrast to these and every other lib I've worked with, Swim is abstract.

I start wondering if I'm too dumb or otherwise incapable of reasoning abstracting to understand the Async part of Rust's ecosystem.


Thank you for the feedback! We could definitely improve the documentation and we will expand on the "why?"; we will get an updated readme up that expands on this in more detail. Could you please tell me which part of the website clearly explained it to you such that you understood the value proposition?


I am someone who initially really struggled with understanding whatever the Kafka team thinks their explanation is. Even after working on streams and pub sub messaging I still struggle with their explanation.

However, I completely resonated and understood what you guys have built. We actually built something very similar but it was over websockets with protobuf3 just to emulate this exact behaviour. We we're using it for IoT-esque devices which send their data constantly but we needed to keep track and update certain fields when devices would connect and disconnect.

We'll be open sourcing our implementation soon. I will happily want to rebuild it with this, and maybe add a sprinkle of protobuf3 to make sure messages are type safe.


I think the examples could answer that question. They’re linked at the end of the explanation in the README. However, I think each example ought to have a README.

Here’s the first one I looked at. I’m wondering when to use an event instead of a command. https://github.com/swimos/swim-rust/tree/main/example_apps/v...


Many of the example applications are corresponding code to our developer website which contains guides for them: https://www.swimos.org/server/rust/ We will get this updated as it isn't clear


Having to study an example before getting a clear enough idea what to look for definitely narrows down the potential audience. Examples are very useful and important, but, unless they fit in 5-7 lines, they are the second step.

The first step should be a few bullet points listing the key reasons behind the framework and the key benefits it provides.


Maybe stating the obvious but took me an embarrassingly long time to clock this:

SwimOS is an existing streaming framework (Java-based) targeted at full stack / app development.

This announcement is a new Rust SDK for the above - not a new framework.


That is correct. We're working towards replacing our Java implementation with the Rust version and then provide bindings to Java


The Swim Rust SDK contains software framework for building stateful applications that can be interacted with via multiplexed streaming APIs. It is built on top of the Tokio asynchronous runtime and a Tokio runtime is required for any Swim application.

Each application consists of some number of stateful agents, each of which runs as a separate Tokio task and can be individually addressed by a URI. An agent may have both public and private state which can either be held solely in memory or, optionally, in persistent storage. The public state of the agent consists of a number of lanes, analogous to a field in a record. There are multiple kinds of lanes that, for example, lanes containing single values and those containing a map of key-value pairs.

The state of any lane can be observed by establishing a link to it (either from another agent instance or a dedicated client). A established link will push all updates to the state of that lane to the subscriber and will also allow the subscriber to request changes to the state (for lane kinds that support this). Links operate over a web-socket connection and are multiplexed, meaning that links to multiple lanes on the same host can share a single web-socket connection.

There's a number of example applications available here: https://github.com/swimos/swim-rust/tree/main/example_apps If you're interested in getting started with it a developer guide is available here https://www.swimos.org/server/rust/developer-guide/ as well as reference documentation here https://www.swimos.org/server/rust/

Previously on HN:

- SwimOS: Distributed platform for building stateful, real-time streaming apps https://news.ycombinator.com/item?id=22920764

- Real-time traffic light status in Palo Alto powered by swim.ai https://news.ycombinator.com/item?id=19234286


How is it real time if you are using tokio? What latency and deadline guarantees are you able to provide?


"real time" as in figma multiplayer not as in control system for a car (I'm assuming, never used SwimOS)


Doesn't run on bare metal though?


It doesn't unfortunately but this is something that we have discussed internally about targeting. We would be interested to hear about your use-case for it!


I'm just picky about maintaining purity of the term "OS".

If "OS" can include WebKit and Lua and anything with a file API and an interpreter then "OS" just means "an abstraction" and I don't need two words for the same thing

Like, Chrome is not an OS, no matter how big a bonus someone at Google got for that clever bit of marketing.


Why do so many products that aren't an operating system calling themselves WhateverOS?


Looks very similar to Akka, where agent is actor and Lifecycle is Behaviors.

An interesting project for a complex task - streaming stateful processing.


This is super cool!! Back many years ago now I had a project that dealt with realtime data + geofences so I used tile38. I'm super curious to check this out!


We hope that it goes well and we would love to hear your feedback!


What about adding differential data flow?


How does this compare to Erlang?


I've worked with Erlang/OTP in the past, although it's been a while since I last used it. I have fond memories of working with it, and I appreciate its unique strengths. One notable similarity between Erlang and SwimOS is the actor model, but their use cases are quite different. Erlang excels at fault tolerance, robust supervision, and lightweight isolated processes, with a focus on message passing (which, if I recall correctly, involves significant data copying underneath).

In contrast, SwimOS is designed for high-performance streaming, boasting a heavily optimized WebSocket implementation that leverages multicast, multiplexing, and differential dataflow (delta encoding) to efficiently handle massive data volumes. SwimOS utilizes zero-cost abstractions and lock-free mechanisms to achieve its goals. Its primary use cases are data observability and real-time collaboration applications, with an emphasis on in-memory processing and recent data. This processing window is similar to a retention window in distributed log technology like Kafka, where the focus is on handling large volumes of data in real-time to enable timely decision-making, without necessarily storing all the data long-term.

While both technologies share some similarities, their design and use cases make them suited for very different problem domains.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: