This is awesome! I've been developing firmware in Rust for a Hall Effect keyboard (with springless, magnetic separation switches!) I'm developing/inventing (just ordered the first PCBs!). I really wish we had something similar in the embedded Rust ecosystem as a lot of the crates mentioned don't support #![no_std] (declaration you make at the top of your Rust code to indicate that the compiler should not include the standard library).
The world of embedded Rust seems to be developing incredibly fast. Especially since there's so much hardware that you can use with it now and more is on the way (especially in the world of RISC-V... Holy crap there's a lot of interesting and exciting hardware coming!). As a result of that I seem to be regularly encountering situations where I try to do something basic but no one has "made a crate for that yet" (that works with #![no_std]).
Just the other day I found out just how difficult it is to benchmark something (i.e. grab the current time, do something, get the time again, then subtract the difference... It's really, really complicated! Way moreso than it should be). If someone wanted to make a name for themselves by developing widely-used software now is the time to make your mark! The world of embedded Rust is waiting for you!
FYI: I'm still a newbie when it comes to Rust but I was able to develop a universal driver (works with any hardware platform) for working with 74HC4067 and 74HC4051 analog multiplexers in just a few days... https://crates.io/crates/analog-multiplexer
There is a section called no-std crates which lists a couple of useful crates like bbqueue and heapless. I find it insightful to browse through the public repos of the authors of popular embedded rust crates and see what crates they use most often.
Yeah, magnetic levitation is the exact opposite of magnetic separation. Magnetic levitation switches are basically just linear switches that get harder to press the farther down they get.
Magnetic separation switches are the opposite: The force is strongest at the start but then gets weaker the farther down the key travel you go. Here's a force curve that I've measured for my 3D printable magnetic separation switch:
Yeah it's just the end of the road, as it were. At the end the keycap, stem, and switch body start to compress so there's some give before the super fun explosion.
In my little bit of searching, most people seem to be saying they like it (many a lot), and most of the dissenting opinions seem to be more about the very low key travel (0.7mm), which is merely something that the maglev keyboard made possible and more bearable, rather than something fundamental about magnetic switches or movements (whichever it is).
(I’ve never seen one of these keyboards or even heard of them before, and am thus speaking from an uninformed position.)
Sure, it's a 3D printable key switch design (yes, it's smooth--not scratchy). I've already made one hand-wired prototype and I posted about it on Reddit here:
I've also designed two keyboard stablizers: One that's "regular" (Cherry compatible) and one that's meant to work with my hall effect switch design that incorporates magnets to ensure it never rattles. I haven't released them yet though (Soon™).
My key switch design was created in OpenSCAD and is parametric. So if you want you can customize every little detail of the switch from the size of the magnets to the travel to the initial force required to depress a key (by controlling how close the magnets get at rest).
Once I have a working keyboard firmware I'll release the whole thing open source so everyone can play around with it.
Oh! I also made a keycap design library for OpenSCAD to go along with all of this called the "Keycap Playground"...
That video is so frustrating: I’m just trying to look at the thing, but your LEDs keep on cycling through annoying and distracting patterns that quite ruin the picture by throwing the camera’s light metering off and adding lens flare. I got about halfway through before it was altogether too much for me.
I would have really enjoyed watching the same video with the LEDs turned off.
Anyway, I’m going to look more closely into this now, because other that that it looks like a really interesting project! :-)
1) For error handling, I think anyhow and thiserror have been gaining popularity for applications and libraries, respectively. There's also eyre, a fork of anyhow with some more options apparently, and color-eyre, which is a variation of eyre for colorful backtraces. Long story short, yeah, there's a lot of crates.
2) I've heard good things about sqlx as an alternative to diesel that lets you write raw type-safe SQL into your code and get it checked at compile time. It has good async support too.
3) I would probably mention async-std as an alternative to tokio.
Thanks for mentioning SQLx. I’ve been writing a SQLite integration recently using rusqlite, which is great and does the job, but some of the SQLx features, like the checked SQL statements and being able to fetch directly into a struct, look like they are worth checking out.
> Long story short, yeah, there's a lot of crates.
And I'd add one more to the pile: my [SNAFU] crate. It actually does what the author says they do by hand:
> So, I just write the boilerplate and make my errors descriptive enough I don’t need a backtrace. When I want to get fancy I implement the built-in `Error` trait, which used to be kinda useless but is now more helpful. And in another five years it’ll still work just fine.
SNAFU is all about creating enums for each error case so that the hierarchy of types tells you exactly what and where the error came from. In my ideal usage, you don't need a stacktrace or even origin line number / file name because each specific combination of error is used only once.
The nesting of types within specific contexts gives what I like to call a "semantic backtrace"
Example usage:
#[derive(Debug, Snafu)]
enum Error {
#[snafu(display("Unable to read configuration from {}: {}", path.display(), source))]
ReadConfiguration { source: io::Error, path: PathBuf },
#[snafu(display("Unable to write configuration to {}: {}", path.display(), source))]
WriteConfiguration { source: io::Error, path: PathBuf },
}
type Result<T, E = Error> = std::result::Result<T, E>;
fn example(path: &Path) -> Result<()> {
let data = fs::read_to_string(path).context(ReadConfiguration { path })?;
fs::write(data, path).context(WriteConfiguration { path })?;
Ok(())
}
This creates the `Error` enum and implements `std::error::Error` for it, plus what I call context selectors to easily add on to fallible calls to help group them. Note that both SNAFU errors are caused by an `io::Error`, but the context in which the `io::Error` is created is vastly different.
> for applications and libraries, respectively
I've yet to understand why people want to draw a line between these two cases. For me, the only difference is that the error type in a library needs to be conscious that it's part of the public API. That's why SNAFU can generate [opaque] error types
---
Other bonuses:
- Doesn't require heap allocation (but can make of use it)
- Works in no-std environments
- Comes with extension traits for `Result` and `Option`, and feature flagged ones for `Future` and `Stream`
- Supports generic errors (types and lifetimes)
- Can create backtraces (either from the backtrace crate or the unstable `Backtrace` type in the standard library)
once_cell allows you more ways to do lazy initialization.
Like by using `once_cell::sync::OnceCell` instead of `once_cell::sync::Lazy` you can determine the init function when trying to access the once cell. For many cases `Lazy` is what you want but for some `OnceCell` makes it just so much easier.
And then you also can use the types outside of a "lazy initialized static/global" use case.
As a newer Rust developer, this is a very helpful list—trying to find appropriate crates has really been a crap shoot. One thing I can note from personal experience, though, is that saying `structopt` “is relatively slow to compile and doesn’t produce the tiniest code” is quite an understatement—its binary overhead is 379.6KiB and compile time is over 15 seconds[0]! Parsing args is a tiny part of an application, so giving over so much time and memory is crazy to me. I’ve personally settled on using pico, which doesn’t use proc macros so you don’t get the sugar of a declarative arguments object, but it’s fast to build and almost as small as the lightweight recommendation `argh`.
While structopt may be too much for many applications, I would argue it's not necessarily as overkill as it looks. For many CLI applications, arguments parsing and (relevant) help are their primary UI. It absolutely makes sense for larger applications to put a lot of effort into making the experience as painless as possible.
Of course if an application just has a few simple args then it may not be worth it. But for something like, for example, ripgrep[0] you'll want something that produces nice help and context sensitive error messages to help the user find the commands they want.
In short, a good UI can be a complex beast even if it's only a text interface.
I get how important UX is, and I still struggle to fathom what in structopt even generates 20x more machine code and 30x slower compilation versus a simpler argument parser which can do 95% of the job. Someone at some time thought that every line of code had to be there for some reason, but it just doesn’t seem like a reasonable default choice to me for most apps, because it’s a huge penalty just to handle some edge cases that won’t come up most of the time.
ripgrep, your example of something demands a complex arguments parsing system, seems to have had problems with unnecessary changes in clap (on which structopt is based) slowing down compilation for no reason[0][1] and currently uses none of its configurable features except suggestions[2].
Which provides a interface like the std library but async.
(Through tokio copied a lot of this by now).
It uses the same execution implementation as `smol` and generally most of the code from it is shared with `smol` through internal dependencies like `async-executor`.
Through lie tokio it's a bit bigger then `smol` which affects compiler times.
I'm learning rust and tokio vs async-std confuses me: if I have a library that uses tokio (e.g. tonic grpc), can I use a library that uses async-std and mix calls to functions of those two libraries in the same function?
There is also a `tokio02` feature which let's you run tokio inside of async-std.
The problems are:
- The global `spawn` method need to know on which executor to spwan thinks, the ones for tokio look for a tokio executor the ones for `async-std` for a async std one.
- The async-io implementations might require to be run in the executor they are shipped with.
- Some combinators shipped with some runtimes also need to be run in that runtime.
This is a bit annoying. I believe everyone involved would have preferred a single general purpose runtime, but to all kinds of reasons this didn't work out.
The original plan was to provide generic interfaces in the futures library so that most code (including things like grpc) can be independent of the runtime.
But people couldn't agree on a common interface. The main problem was around how to do the `AsyncIo` trait and timers.
Since the introduction of async/await the runtimes have converged a lot more in their external interfaces, and there are plan to put `AsyncRead`/`AsyncWrite` traits into std.
After this the divergence problem might maybe be fixed slowly.
---
To provide some more information about the problem:
Basically a executor consists of three parts:
1. The thing driving/running/polling futures.
2. A reactor handling the async io details.
3. Something to handle things lie `Delay`/`Timeout`.
For the first point the problem is a missing standardized way to `spawn` new independent futures and another way to do so locally (`LocalExecutor`/`LocalSet`).
For the second point it's not so likely that things will converge but we could agree on a way to have multiple `Reactors` running in parallel or similar in ways which should "just work" without to much overhead.
For the third point people currently can't agree one specific implementation as far as I remember once they do we might at least have a generic timer interface.
Anyway by now most of the (new) external APIs of both async-std and tokio now mirror std (tokio copied that once it turned out it was a good idea).
This also means that most implementations can be made to work with any executor by having a feature for any of the executors and then depending on the feature import different things, for most parts most of the code will just work by importing think from async-std instead of think from tokio.
There’s no clean async abstraction later; once you pick a side, you’re stuck with crates that support it.
Some popular crates let you pick using flags which one you want when you add them... but, tldr: no.
It’s not ideal, but it’s very flexible.
Given they weren’t sure exactly what the async runtime should look like, and the strong backwards compatibility promise, they opted for “get something people can use for now”.
...of course, the python packaging story shows this is a stupid idea in the long run, and I believe the plan is to consolidate into a single “std” async runtime eventually.
...but for now, in your situation, you’re basically screwed I’m afraid. Fork the crate and migrate it yourself is probably the only real solution.
The downside is that doing that usually unearths dependencies that are also using the wrong runtime, etc.
That's not fully correct. A lot of futures will work independent of the runtime. E.g. all the sync primitives/channels/etc from all sources (futures-rs/tokio/async-std/futures-intrusive/etc) will work with all runtimes.
Then a lot of objects which require a runtime (e.g. an async socket) might still work on a foreign runtime as long as the original runtime is running. E.g. I think you can use async-std sockets from inside tokio, as long as the async-std runtime is also running in the background. Same is true for the other direction to a certain extent, but it's a bit more complicated to set up.
So overall you will need to understand how the runtimes work and if particular pieces can be mixed and matched. If you don't want to invest in that knowledge => stay on one side.
And before someone starts to blame the Rust ecosystem for that: It's not really different in any other environment. You can't easily mix and match GTK, QT, libuv and boost asio eventloops. It's event a lot harder than trying to combine Rusts async types.
> So overall you will need to understand how the runtimes work and if particular pieces can be mixed and matched...
Hm... really?
I was under the impression that it was more correct to say that mixing runtimes is undefined behaviour?
It may work, but just like mutable aliasing using unsafe may not cause memory corruption; it just happens to work in the current implementation, on the current platform, on the current version of rust.
There is no guarantee it will continue to work in the future.
Maybe I misunderstood, but I thought it was a lot more complicated than just starting multiple events loops and you’re fine.
Yes - really! It would be undefined behavior if the runtime doesn't make any promises about it. But some do make the promises that it will work. I think both async-std and tokio make sure IO related futures work from tasks that are driven by a different runtime - otherwise stuff like `future::executor::block_on(task)` would also not work, and neither would the blocking wrappers around some of the HTTP clients.
However the IO objects might need to be initialized within a certain context (e.g. some TLS variables need to be set for tokio). And there might be a few different things one need watch out for. E.g. if you stop the background runtime things might get weird - however this e.g. can't happen with async-std since the background runtime will never shut down.
The problem is not the executor of the futures itself.
But the "Reactor" which is needed to do async-io and the "TimerScheduler" which handle thinks like `.delay(..)` or
non os-native timeouts.
What you often can do is having a `Reactor` and time sheduler running for all runtimes involved.
This e.g. works very well with async-std due to it's simple design.
But for tokio it's more complex as you need to provide the reactor and some other hints through something comparable to thread local variables (instead of globals).
This is where the `tokio02` compatibility feature comes in which makes sure to provide access to tokios reactor in the async-std future executors.
Another problem is the global `spawn` method. You normally don't want to run two future executors but as long as their is no abstraction layout over spawn you will have to. Furthermore tokio does a lot of fancy things which you likely will never see in a generic API which is another problem.
For example normally if you want to do blocking code you spawn this in a thread pool to not hinder other async code. But in tokio there is a way to "overlap" the pool of blocking and non-blocking code, basically up to n worker threads of the non-blocking thread pool can be marked at blocking at a time. This is one of many thinks which increase complexity which other runtimes like async std avoided due to it not being worth the additional complexity in >99% of the cases.
Except that tokio is to some degree written for that 1% of cases where all that additional complexity is needed. Because a lot of dev time come from people which work for a company which does need it for their product.
I can probably. I guess a few prinicpal/staff engineers can too. But doing cross FFI async function invocations and trying to unify runtimes in that fashion is not a thing that people usually do. And it still wouldn't allow you to call QT socket APIs and Netty socket APIs from the same thread.
You missed the point, the languages I mentioned have it as relevant enough to have the async runtime as part of the standard library, instead of leaving it to 3rd parties.
C++ coroutines are not more „part of the standard library“ than rusts async support. One might say it might be less, since they arrived so late in c++ lifetime and more Production libraries have been built without them und mind.
And again, they don’t make GTK and Qt and boost Asia eventloops interoperable. Even if you implement co_await support for each of them individually
Like any ISO standard, certain details are expected to be implementation dependent in any ISO C++20 compliant compiler, not missing like on Rust's case.
async-std tried to mirror rust's libstd, and tokio adapted it.
Similar the global `spawn` and the value returned from it work kinda the same +- some naming differences.
Even the `LocalExecutor`/`LocalSet` can be abstracted over to erase their differences.
That also true for all simple usages of timout.
The remaining differences are:
- different internal implementation details
- naming differences which are not that easy to abstract away
- some detail about time/timeout handling I forgot which for many use-cases doesn't matter
- the AsyncRead/AsyncWrite traits but this is going to get into std so this will go away soon
- some "advanced" features like the fact that tokio overlaps the non-blocking executor thread pool with a blocking worker thread pool, but that in many case not a very important feature.
- some differences in e.g. how to specify the number executor threads etc.
Most other differences are "old legacy left overs" from the pre-async/await futures time periode.
So while a generic in-language abstraction likely won't happen anytime soone one which uses feature flags and re-exports as basis should be very doable in the close future.
Same thing happened in Ocaml land w.r.t. async vs lwt. Not sure what the current situation is but people had to abstract their libraries and do 2 implementations to support them.
I read through this and have to say I disagree with several of the points. Rather than go through them all one by one, I'll just say that my experience with several of these create differ greatly (tokio in particular), and several of the crates recommended would not be the ones I suggested. The http section also doesn't mention Actix or Tide or Rocket. Then again, I am relatively new to Rust myself (1 year), so take this with a grain of salt.
There are a couple of key differences between the two ecosystems here which affect the tradeoffs:
1. Rust's cargo tool manages dependencies, it's almost universally used, and it does a good job. It's most similar to a mature Pipenv.
2. The Rust community normally does a good job with semver and the type checker makes it much easier to notice breaking API changes.
3. The Rust standard library has a strict policy of not breaking backwards compatibility.
4. There are tools like cargo-audit, cargo-geiger, cargo-crev and cargo-deny that can be used to enforce various policies across all dependencies.
5. Rust allows building single-file executables, so you don't need to distribute dependencies at runtime.
So in practice, people tend to standardize one or two good solutions outside of the standard library, and very occasionally the best ideas will make it into the standard library. It's partly a philosophical difference, but the entire ecosystem of tools works together to support that philosophy.
Not a rustacean myself, but I think the key issue is that Python the language predates PyPI and the whole modern mentality of it being trivial to pull in lots of third party stuff, so the expectation was that the standard library needed to be "batteries included" for the language's big initial use-cases.
So part of it is that Crates has always been a first-class part of the Rust ecosystem, but the other piece is observing from Python what happens when some key functionality in the stdlib ends up completely stagnated because it can only ever add features or fix bugs with a new language release (urllib/urllib2/urllib3 vs requests, asyncio vs trio, etc). So Rust prioritizes stabilizing the kinds of interfaces and language features which allow third party functionality to plug together in a reasonable way, rather than committing to supplying all of in the core.
Yes, it is a purposeful difference in philisophy. Python has challenges from the fact that its standard library is big - the interfaces (and so design) for a lot of these things are locked in place and you get stuck with suboptimal situations. And a lot of maintenance is required by the core team. Python 3 used the opportunity of backwards incompatibility to fix the worst issues, but that's a one time opportunity.
Rust consciously decided to keep the standard library minimal due to the backward compatibility guarantees they want to give, and then provide an excellent packaging story so that it's easy to pull in more capabilities. This allows for much faster iteration and improvement on what Python might consider 'core' capabilities.
Lots of the modules provided here are not the first attempt to achieve what they're doing. They're in the list because (the author considers them) the best. If they were in the standard library, maybe we'd be stuck with a worse version?
Rust promises to never break or remove anything from the standard library, and Rust also wants to be a language usable for the next 40 years. This means that it has to be careful not to add stuff that's going to become outdated and deprecated cruft that they'll have to maintain forever.
Rust has already made a mistake of including a channel implementation in stdlib, which ended up being inferior to crossbeam-channel. Now, as you can even see from TFA, there's a dilemma whether to use the slower, less flexible one from stdlib, or a crate. If there's only a crate, there's no dilemma :)
Yes, it is a difference in philosophy. Here is a blog post from 3 years ago where the libs team explored making an official set of libs to more closely match a "batteries included" experience: https://blog.rust-lang.org/2017/05/05/libz-blitz.html
This effort did not end up succeeding. Instead, a more organic, informal process seems to be working pretty well. Lists like this seem to be useful, even if none is officially blessed.
A few "two cents"/opinions from me, surely not the complete picture.
External crates are easy to include and often come also from rust core developers, this does not actually slows one down but allows for much more flexibility.
The big benefit over python is that rust does not dictates what you need to use.
And with the trait system you can often just plug in another, now maybe better fitting, solution for a specific crate.
Further, note that rust is normally shipped to the user compiled to a binary with all rust code statically linked in, while python isn't. Python is shipped often as code, and if the std library would not include that much that'd mean that one had to ship lots of module code along.
So, rust would not actually profit from having a big standard library the same way Python does.
Having a big standard library makes it also harder to deprecate and remove cruft (<https://lwn.net/Articles/790677/>), which can get a problem over the long time.
In addition to the issues other people have mentioned, Python benefits from a large standard library in a way that Rust doesn't, because Python scripts don't require a build step.
So it's easier to use Python than Rust for a little bit of "glue" in a place where maintaining a pile of dependencies would be inconvenient.
If you can just stick a single .py file in a git repo somewhere and arrange for it to be run, that's a big gain compared to having to maintain a list of dependencies and arrange for them to be installed and so on. So it makes a big difference that you have access to regular expressions and tar files and whatever in that situation.
stdlib of any language is a place where libraries go to die.
i agree it's surprising but it makes sense - cargo works very well and some packages are created by the same team that develops the stdlib, except they're free to break the api once in a while as needed.
In Rust this is only guaranteed to be true of "tier 1" targets. Rust also supports many "tier 2" and lower targets where the guarantee is only that the standard library will build (not that it actually works) and even then on some platforms only the core is supported.
yeah that was the point - less guarantees that something works forever means less maintenance. in the current world the balance is with leaner stdlibs because discovery and acquisition of external libraries is so simple.
This has been one of the biggest pain points for me when using Rust, and I think this is a bad philosophy.
Leaving the implementation of basic functionality, that everyone needs in any language if you're trying to do anything even slightly non-trivial, to the "organic process of letting things naturally arise from the community", has so far been a very effective way to ensure no packages can naturally talk to each other ever, or have any composability.
Everyone re-implements their own "personal abstractions library of crap I need often" like we used to do with C because package management in C is horrid and the stdlib is as crude as it comes, or use one of the many crates of people who decided to publish their 10 LOC crate, and then you get a proliferation of microscopic packages that implement trivial functionality and the ecosystem starts looking a lot like NPM.
Rust could really use some more top-down leadership. Say what you want about Rob Pike (and I'm not saying that Golang doesn't have pain-points with the leadership), but almost everything in the ecosystem composes like magic.
On top of that, everytime I try to use Rust I'll need a feature and will run into some issue from 8 years ago, where everybody admits it's a problem, there are good and obvious solutions on the table, but all those 8 years were spent circlejerking and bikeshedding and no one can make up their minds; it's not even a "this is what the future of this issue looks like but it will take some time", it's more like a "after 8 years consensus hasn't been reached and no progress has been made on something a lot of people agree is a problem".
Sometimes the need for group consensus serves only to corrupt the conceptual integrity of systems, and dilute any sense of vision into oblivion... I wish we could do without it sometimes
Yep. If a language has too small a standard library, you end up with NPM for JavaScript. Too big a library and you end up with C++. Can’t please everyone...
> [Things I avoid] parking_lot: More compact and efficient implementations of the standard synchronization primitives. [goes on about efficiency...]
lol, really? No, like, not at all.
Parking lot has 1 killer feature, and 1 killer feature only, and this feature makes using the standard synchronization primitives _almost always_ the wrong choice for almost all applications:
DEADLOCK DETECTION
That's right, if you ever need a Mutex, chances are that your program might run into a deadlock at some point. Do your future self a huge favor and don't use the standard library's std::Mutex. Instead, use parking_lot::Mutex, turn on deadlock detection at compile-time, and save your future self potentially hours of trying to figure out what your program is doing, why is it deadlocking, and how to fix it.
parking_lot::Mutex panics if your program deadlocks, printing out the backtrace of each thread involved in the deadlock. This makes discovering, understanding, and fixing dead-locks a 5 minute issue, that's easily and quickly discovered during development in the minutes after introducing a deadlock bug.
I mean, you are completely right that, on top of that, parking_lot synchronization primitives are much much faster than those in the standard library. But that's only the cherry on top.
---
FWIW, nice post, I agree with all your judgements there. If you are looking for an async runtime that's smaller than tokio, check out the `smol` crate. For a thread-pool, i either just use rayon's (you can just push tasks onto its thread pool), or the futures's simple thread pool from the futures crate.
Side note: parking_lot is set to be uplifted into the standard library at some point. However it currently doesn't meet the quality standards of the standard library so more work needs to be done.
That's interesting; I imagine that they're not going to just wholesale replace std::sync::Mutex and the like with the parking lot versions due to the fact that at least last time I checked, doing that would cause breaking API changes (e.g. Mutex::lock not returning a Result, so are they planning on just making a separate module with the parking lot API as an alternative? (I supposed they could just make the parking lot Mutex return a Result that always returns Ok, but that doesn't really seem like it would appeal to most people).
Hopefully they make it a breaking change in a new edition. A little pain when updating will be much better than carrying two slightly different implementations in the long run
Cool, I didn't even know about the detection feature. When I get a deadlock, I attach a debugger and look at backtraces. So far that has always been enough for me. The nice thing is that the deadlocked threads aren't going anywhere, and wait patiently for me to inspect them :)
> The nice thing is that the deadlocked threads aren't going anywhere, and wait patiently for me to inspect them
Same opinion here. Both deadlocks and null pointer errors are fairly easy to debug. You can easily observe when exactly the happen, and the stack-trace typically delivers the answer what went wrong.
Other kind of race conditions like any memory corruptions can be such much worse to figure out.
That depends though. In Java/C# dereferencing a null reference is a runtime error and you can pinpoint exactly where it happened. In C++/Rust, dereferencing null pointers is undefined behavior since the compiler makes assumptions that a dereferenced pointer cannot be null.
If you need deadlock detection, probably your high level design is problematic. But I agree this is extremely often the case, so that's extremely often an interesting and valuable tool.
After chasing a bug for hours just to discover that it is a deadlock, its always easier to realize, in hindsight, that deadlock detection would have been useful.
Independently of how perfect your architecture is, the moment you add a Mutex to your app, you are opening the door for that to happen.
More like the moment you add a second Mutex in your app, ad-hoc handling and no policy to prevent deadlocks.
The solutions are well known and I'm simply sad to see them far too little used.
But in the case you insist in doing that (not using architectural solutions, some of which can be checked statically), yes of course runtime deadlock detection is a very valuable tool. I agree that it is even useful if you rely on policy but can't automatically check that they are respected.
I mostly agree with this except I think the bigger thing is that you should rarely be using locks in the first place. Instead use existing parallel data structures and atomics.
> Not aware of any great encoders... Video, I haven’t used enough to have an opinion on.
If you don't mind having a -sys dependency, you can use an FFmpeg binding. There's ffmpeg-next[1] (which is a continuation of the abandoned ffmpeg[2] crate), and a few others I have no experience with.
Very good list. There have recently been similar posts about what crates to choose. Always followed by discussions about some of the choices.
In professional environments where besides the product time to market and costs are the key points, we don't want to spend time on this kind of discussions and decisionmaking. Or taking risks using libs with 0.x versions.
It really is a pitty that the decision to keep essential stuff out of the standard lib is limiting Rusts usage in professional environments. Developer resoureces are another reason, but this is a chicken egg problem. The standard lib issue can be solved much easier.
Would be really interested in a solid tree/list/doubly linked list structure. The borrow checker appeared to make the construction of such a structure impossible.
It's really hard to determine what parts of comp sci are niche a-priori. Most of my current work involves recursive data structures (trees/graphs/linked lists).
I don't mind the callout that you need to use unsafe for recursive structures, but it would be useful to acknowledge that as part of the language design in the rust book. As a beginner you can easily pick up a simple LeeCode problem to learn the syntax and accidentally pick an impossible problem.
For the beginner there's "Learn Rust With Entirely Too Many Linked Lists"[0]. The main issue I have with them is they tend to optimize poorly on modern hardware even when they theoretically should be the right data structure to use (IMHO of course).
Yes, it is. It's one of the tough parts of the book, I cannot assume anything about the audience. That includes "they even know about recursive data structures," let alone "they want to write them."
In general, it's not possible to enumerate all negatives, but it is for all positives, so I try to say more about what is than what is not.
The issue is not graphs, trees, and lists. These are useful! It's implementing them the way that you do in CS 101 that is the issue.
First off, thank you for all of the work you've put into Rust! I think the challenge that you face is that if the problem is CS101 courses, then Rust will always be unfamiliar to the majority of practicing professionals.
Cyclic/recursive datastructures are common in web services, where one needs to model bidirectional relationships ( just think of friends on a social graph ). As well as several primitive datastructures such as the O(1) LRU cache.
Checking the Rust docs out, a dev who encounters this problem for the first time will eventually need
I'm a little biased on this front, when I first picked up Rust I chose to tackle an O(1) LRU cache as a quick learning exercise. I ended up beating my head against a wall for 20 hours. However on my current project I'm analyzing the wikipedia dataset ( another bidirectional structure ).
It would be useful to have a canonical/standard way to address such programming tasks other than "don't" - particulary if the way to address them is substantially different from what someone would have learned in a different language.
Do you really think the majority of professionals have CS degrees? I’m not so sure. And beyond that, it would have to be “a majority with a degree” and “a majority that choose to implement data structures as an initial/early task.” I’m really not convinced the numbers actually play out that way.
Hmm playing off my own anecdote - I do not have a formal CS degree, but I ended up taking courses and reading a few of the typical books later on.
I think while my early experience biased me, the pain would have happened at one point or another eventually. I'd be surprised if the majority of web devs, and data science folks exploring rust wouldn't hit this problem at some point or another. As an example, one could easily run into this problem while writing an wiki,instagram, or twitter clone!
Sounds like we're opposing anecdotes; I do have a CS degree, but didn't try to start with data structures.
You may want to use something with those characteristics, sure, but that doesn't mean most users start by trying to implement, rather than use something existing.
It’s not clear what would make it easier, and it’s also not something people are particularly crying out for. They’re not super common structures and as soon as someone makes a crate, you can use it and get on with life. And if you have to, you can just write what you would in C, so it’s not harder exactly, either.
I'd +1 a quick section in the book on when unsafe is the best choice using some referential datastructure ( along with potentially a few other examples where unsafe must be used )
>I don't mind the callout that you need to use unsafe for recursive structures
You don't, really. Recursion is not the important part. The ownership model is hierarchical. If your structure is a directed tree with all links (pointers) flowing out from the root, it can easily be built. As soon have you have cycles or multiple parents, you'll need either unsafe or some ownership extension like shared ownership (Rc/Arc).
Singly linked list => Easy
Tree => Easy
Tree with backreferences => probably best to use unsafe
Graph => probably needs unsafe for best performance
The post you are replying to says "efficiently implemented with unsafe". For a linked list, you generally get (has unsafe code + efficient) or (no unsafe code + not as efficient).
I like the Linked-list crate [0] from contain.rs [1] (a collection of Rust datastructure) as it has a cursor which is the thing I miss in the std implementation.
For thread pools I'm pretty sure you can use Rayon.
I agree with 99% of this but I can't agree with Crossbeam. I use it basically every time I need multiple threads because you always end up needing `select!()`. As far as I know std doesn't have anything like that which makes its channels rather useless.
While I agree with your sentiment, and do the same, `select!` is not more than looping through a couple of `try_recv` calls. So in the spirit of the author, a few lines of boilerplate vs a new dependency.
> `select!` is not more than looping through a couple of `try_recv` calls.
That's wrong. Any `try_recv` call will not make the thread park until one of the expected action happened. You can implement a busy loop which calls `try_recv()` - but this is by far not as efficient as what `select` does.
C# has a pretty huge standard library (or Microsoft official packages) that cover a lot of things here. Quickly running through the rust list, the things I suppose I would add:
- Like the author I don't know much about this topic, but for hashing BCrypt is pretty well known and appears well respected (https://github.com/BcryptNet/bcrypt.net)
- Microsoft have just added their own internal JSON serializer to .NET Core, so you should probably use that where you can, but it doesn't have anywhere near the same features that Newtonsoft's Json.NET does, so if you're in need of those then that's still the battle-tested standard (https://github.com/JamesNK/Newtonsoft.Json).
Time:
- Personally I've never had a reason to use it over the standard DateTime, but I've heard good things about NodaTime (https://github.com/nodatime/nodatime)
Images:
- System.Drawing still exists in the standard library for Core, but it's Windows only I believe
- ImageSharp is a great library for native image stuff in .NET Core, works across all platforms (https://docs.sixlabors.com/index.html). There was some hullaballoo a little while back about it's license, but that appears to be resolved now as it's Apache 2.
- .NET obviously has it's own httpclient, but it's worth mentioning Polly here (https://github.com/App-vNext/Polly). It allows you to extend the default Http Client with short circuits, automatic retries, exponential backoffs, etc.
Databases:
- Probably doesn't need saying, but Entity Framework for an all in one migration, entity mapping and database access library (https://github.com/dotnet/efcore). Since moving to Core, it's also a lot faster now, although I think it's still lacking a couple of features from the old framework version.
> - Like the author I don't know much about this topic, but for hashing BCrypt is pretty well known and appears well respected (https://github.com/BcryptNet/bcrypt.net)
bcrypt is a "Password hashing function", not a cryptographic hash. bcrypt has tunable performance, it's not particularly fast to start but can be made extremely slow (to thwart brute-forcing of weak passwords). It also takes multiple inputs (passphrase, salt, parameters) vs 1 parameter for a cryptographic hash (data to hash). For a cryptographic hash you probably want SHA2, Blake2, or Blake3.
The world of embedded Rust seems to be developing incredibly fast. Especially since there's so much hardware that you can use with it now and more is on the way (especially in the world of RISC-V... Holy crap there's a lot of interesting and exciting hardware coming!). As a result of that I seem to be regularly encountering situations where I try to do something basic but no one has "made a crate for that yet" (that works with #![no_std]).
Just the other day I found out just how difficult it is to benchmark something (i.e. grab the current time, do something, get the time again, then subtract the difference... It's really, really complicated! Way moreso than it should be). If someone wanted to make a name for themselves by developing widely-used software now is the time to make your mark! The world of embedded Rust is waiting for you!
FYI: I'm still a newbie when it comes to Rust but I was able to develop a universal driver (works with any hardware platform) for working with 74HC4067 and 74HC4051 analog multiplexers in just a few days... https://crates.io/crates/analog-multiplexer