I feel like rust has some good sweet spots right now. I care about these but maybe not everyone else does.
- Parsing untrusted inputs. nom[1] is a joy to use, and lets you kind of effortlessly and fearlessly process random input from wherever and turn it into useful data structures. If your data is very regular or in a standard format, then serde[3] is very hard to beat if it just boils down to 'derive(Deserialize, Serialize)' on your Rust struct.
- Bulk data processing. rayon[2] makes pegging your machine easy if you have a lot of work do to, and the Rust semantics around thread safety, data ownership, and explicit copying make it kind of trivial to reason about how your data gets from input to output and tuning it to be crazy fast.
- Generic systems language. Maybe this one is personal, but I find it's more productive to write generic cli applications and whatnot in Rust over C, ruby, or python. There are some nice libs to make part of this pleasant (structopt[4]) but this really boils down to reliability. Because Rust makes it obvious where things can fail so I can deal with it I have way higher 'just works' outcomes in Rust than other languages. I might spend slightly more time making it compile, but I spend basically zero time debugging runtime failures and this is kind of indescribably amazing.
We built a new project in all "modern C++". It is 100% shared_ptr, unique_ptr, std::string, RAII, etc. It initially targeted C++17 specifically to get all the "modern C++" goodness.
It segfaults. It segfaults all the time. It is entirely routine for us to run a new build through the CI process and find segfaults. We fuzz it and find dozens of segfaults. Segfaults because of uninitialized memory. Segfaults because dereferencing pointers. Segfaults because running off the end of arrays. Segfaults because trusting input from the outside world ("the length of this payload is X bytes").
This is where the "modern C++" people tell me we must be doing it wrong. But the reality is that "modern C++" isn't as safe or as foolproof as the advocates say it is. But don't take my word for it - this whole thread is about Google people coming to the same conclusion.
Meanwhile I can throw a new dev at Rust and watch them go from zero to works in a week or so, and their code doesn't segfault, doesn't panic, and actually does what it is supposed to do the first time. Code reviews are easy because I don't have to ponder the memory safety and correctness of every line of code. Reasoning about unwrap() is trivial. Finding unsafe {} is trivial (and removing it is also usually easy).
I too used to program in C++. Every Monday morning, it was the same routine: as I enter the office, the stench of decaying bodies is overwhelming. Yet I must gather my strength to collect and identify the people killed over the week-end by C++ memory safety issues gone unchecked. Once that's done, I start my daily Scrum standup at 11 and I start coding a bit. First build at 11.30, first segfault at 11.35. Then it's pretty much the same routine, after lunch I read the Valgrind and ASan reports, spotting which one of the hundreds of new safety issues it identified might be an easy fix. I go back home riding my bicycle around 7pm, making sure to avoid the cars trying to crash against me due to segfaults. Sometimes I cry at night thinking about all that.
And then one day I found Rust, and all those problems went away. I can now write fearless code, and I don't have to endure the stench of rotting bodies anymore.
I have to say, Rust is looking more mature lately. I wrote a little RSS reader in Rust two years ago, and it was a pain to get all the library version dependencies lined up. Yesterday I recompiled it. No more need for version pinning or Github references; it just worked with a default cargo.toml file. Two years ago there was too much "only works in nightly" or "you need to use this version of that library". Progress.
Any progress on a C++ to Rust converter? Not a "transpiler". Something with enough smarts to figure out when to use native Rust arrays, not "offsets" to imitate pointer arithmetic. I'm surprised that one of the big C++ users, like Google, doesn't have a group doing that.
Something like the cxx crate[1]? You specify your shared objects between C++ and Rust, and it spits out code for both sides.
The guy who maintains it said in the reddit thread[2] about this same topic that the Google people have been sending him good PRs, which is presumably related to integrating Rust into Chrome.
There are actually several of the OpenBSD devs that have been writing Rust lately. Some are doing it for work, some from personal interest. Opinions vary on how much people like it.
Rust and OpenBSD share a lot of technical values though so maybe this isn't surprising. Rust values safety and correctness. OpenBSD values safety and correctness. Rust values 'Just Works' with all the nice things cargo does and the whole 'fearless refactoring / concurrency' thing. OpenBSD values 'Just Works' with sane defaults and the 'batteries included' base system.
Nothing wrong with the OpenBSD people writing in Rust, that's a step away from tossing out say 'grep' or 'ls' and replacing it with something that isn't 100% compatible. That will cause a lot of stuff to break in hard to predict ways. It's like kicking out a random foundation stone from a building and expecting no consequences.
Sorry if you interpreted my comment as advocating for tossing out grep or ls - that certainly wasn't what I was getting at. As one of the OpenBSD people that happens to quite like Rust, I'm aware of some of the other devs who have been using it, and we've talked a bit about Rust in general on occasion. The things that people appreciate about Rust tend to align with the things they like about OpenBSD, because the two projects have similar values. That's all I meant.
At my work we recently went through a large exercise to decide on a common data storage format. The contenders were JSON, MessagePack, and Avro. MessagePack won because:
- Msgpack serialization and deserialization is very fast in many languages - often 100x faster then JSON
- Msgpack natively supports encoding binary data
- Msgpack has type extensions, making it trivial to represent common types in an efficient way (eg. IPv4 address, timestamps)
- Msgpack has good libraries available in many languages
If you do not care about those things (no binary data, no need for extended types, not performance critical) then JSON is just fine.
FlatBuffers, Protobuf, Cap'n Proto, etc., all require an external schema configuration that you compile into a code chunk that you include into your program. Without this it is impossible to make sense of the data. In our case, the data is semi-structured and changes frequently. The prospect of maintaining a schema registry for all the data users and keeping everyone up to date and backwards compatible is enough of a burden that it was excluded.
Avro also uses schemas, but since the schema is embedded in the file it is self-describing so the reader does not need to do anything special to interpret the data. But Avro's C library is buggy and the python deserialization performance was terrible, so Avro was not selected.
Please, no. My experience with the scons build cache is that it exists solely to introduce mysterious compile failures when the cache is out of sync. Eventually you get so distrustful of it that you end up just blowing away the cache as a matter of course.
That usually means your build graph and target signatures are incomplete and you need to figure out where you have a dependency that you haven’t told SCons about.
Exactly. Another way to say this is that if you have an incomplete build graph or target signatures, you will find out from your mysteriously corrupt build output.
I have exactly the opposite experience: I do not want to write a program to compile my program.
Having used both scons and waf, and also having used make, cmake and cargo, I would take any of the last three over scons or waf in a heartbeat. I think cargo is far and away the nicest to use because there is very little to actually do to build your program. Using waf or scons was always an exercise of 'I added a header file, so now I need to figure out the scons/waf API in order to tell it what to do with it'.
pledge[1] allows a process to promise the kernel that it will restrict itself to a given subset of system calls. So you call pledge() with the set of syscalls you need, and then if your application does something else then it will be killed.
The OpenBSD httpd is privilege separated and chroot()ed, and each component only pledge()s the syscalls it needs. This reduces to just the syscalls needed to do filesystem operations inside the chroot, log, and talk on the internet. In the CGI context, if your application only needs to read / write to stdio to talk to httpd, then you can limit yourself to just stdio, or if it only needs to read files then it can limit itself to just those syscalls. If your application does something outside of your pledge() (eg. exec(), because it got pwned) then the kernel will kill it.
Keep reading. The upgrade guide has two parts: the first part that assumes the common configuration (you have console access and are using the OpenBSD boot loader), and the second part[1] in the event that you do not. If you have console access and can boot from iso, you can also use the cd64.iso image to boot into the install kernel and follow the common upgrade procedure on the console.
There is an in-between case where you have a console, but do not control the boot loader (so cannot boot bsd.rd) and cannot boot an iso. It sounds like maybe you are in this situation? In this case, you can still follow the upgrade directions as if you do not have a console[1]. Alternately, sometimes when I am in this situation I just download the install kernel (bsd.rd), move it where the boot loader is hard-coded to look (/bsd), and then reboot. The boot loader will boot the install kernel and you can follow the usual / common upgrade procedure on the console.
There is also autoinstall[2], which can automate the upgrade procedure for you and reduces upgrades to just rebooting into bsd.rd and waiting a bit. There is a bit of effort to create the response file, etc., so this may be overkill for a single instance but is very useful for upgrading fleets of machines quickly.
It depends on the function. Many things will contribute. If your CPU can keep the cookie in cache then loading it repeatedly will relatively fast compared to hitting main memory. If your branch predictor can figure out the jmp over the int3 instructions quickly then that will also be fast. If the function is very short then the retguard stuff will add relatively more instructions to the function so will have a larger impact than if the function was long, etc.
I found that the runtime overhead was about 2% on average, but there are many factors that contribute.
Hey PaXTeam, thanks for having a look! I wrote the implementation, so I can answer some of these.
1. We don't silently skip instrumentation. If we can't find a free register then we will force the frame setup code to the front of the function so we can get one. See the diff in PEI::calculateSaveRestoreBlocks().
2. We do spill the calculated value to the stack. This is unavoidable in many cases (non-leaf functions). It would be an optimization to not do this in leaf functions, but this would also mean finding a register that is unused throughout the function. This turns out to be a small number of functions, so we didn't pursue it for the initial implementation.
3. I'm not sure what you mean by the cookies are shared. Do you just mean that they are all in the openbsd.randomdata section? They have to live somewhere. Being able to read arbitrary memory in the openbsd.randomdata section would leak them, yes, though this doesn't seem to have been a problem for the existing stack canary, which lives in the same section. I see that RAP keeps the cookie on a register, which sounds like a neat idea. I'd be curious to see how you manage to rotate the cookie arbitrarily.
4. I'm glad you like the int3 stuffing. :-) We could always make the int3 sled longer if it turns out these rets are still accessible in gadgets that terminate on the return instruction. Have you found any?
Anyway, I'm happy to see your commentary on this. You guys do some nice work! If you have other suggestions for improvement I'd be happy to hear them. You can email me at mortimer@.
1. both insertReturnProtectorPrologue and insertReturnProtectorEpilogue check hasReturnProtectorTempRegister before proceeding with the instrumentation. so either the changes to calculateSaveRestoreBlocks are not enough to prevent that condition from ever triggering or these checks should be asserts at most or just be eliminated altogether.
2. sure but then this means that RETGUARD is not an improvement over Stackguard/SSP which is not how it's marketed...
3. shared means that entities of a class (threads in a process in userland, every single process/thread in the kernel) see the exact same cookies so leaking a cookie from one entity can allow exploitation by another. this is especially detrimental to the kernel side protection. frequent enough cookie rerandomization can help narrow this channel (RAP has a per-thread cookie in the kernel that is updated on each syscall, and there's some more to reduce infoleaks across kernel stacks, it's all in the presentation).
4. any normal path leading up to the ret is a gadget and int3 stuffing does nothing to prevent that (the underlying logic here is that if one can retarget a return to arbitary addresses then he has already leaked enough information so bypassing the cookie check is a no-brainer too). not only that but in the bsd.mp kernel i just checked, of the 32199 ret (0xc3) bytes only 20236 are actual retn insns, the rest are inside insns. so this int3 stuffing leaves many other instances available. Red Hat tried similar gadget elimination a while ago but noone's using the gcc feature as far as i can tell.
- Parsing untrusted inputs. nom[1] is a joy to use, and lets you kind of effortlessly and fearlessly process random input from wherever and turn it into useful data structures. If your data is very regular or in a standard format, then serde[3] is very hard to beat if it just boils down to 'derive(Deserialize, Serialize)' on your Rust struct.
- Bulk data processing. rayon[2] makes pegging your machine easy if you have a lot of work do to, and the Rust semantics around thread safety, data ownership, and explicit copying make it kind of trivial to reason about how your data gets from input to output and tuning it to be crazy fast.
- Generic systems language. Maybe this one is personal, but I find it's more productive to write generic cli applications and whatnot in Rust over C, ruby, or python. There are some nice libs to make part of this pleasant (structopt[4]) but this really boils down to reliability. Because Rust makes it obvious where things can fail so I can deal with it I have way higher 'just works' outcomes in Rust than other languages. I might spend slightly more time making it compile, but I spend basically zero time debugging runtime failures and this is kind of indescribably amazing.
[1] https://docs.rs/nom/6.1.0/nom/index.html
[2] https://docs.rs/rayon/1.5.0/rayon/
[3] https://serde.rs/
[4] https://docs.rs/structopt/0.3.21/structopt/