Every modern commercial OS is a hybrid architecture these days. Generally subsystems move out of the kernel when performance testing shows the cost isn't too high and there's time/money to do so. Very little moves back in, but it does happen sometimes (e.g. kernel TLS acceleration).
There's not much to say about it because there's never been an actual disagreement in philosophy. Every OS designer knows it's better for stability and development velocity to have code run in userspace and they always did. The word microkernel came from academia, a place where you can get papers published by finding an idea, giving it a name and then taking it to an extreme. So most microkernels trace their lineage back to Mach or similar, but the core ideas of using "servers" linked by some decent RPC system can be found in most every OS. It's only a question of how far you push the concept.
As hardware got faster, one of the ways OS designers used it was to move code out of the kernel. In the 90s Microsoft obtained competitive advantage by having the GUI system run in the kernel, eventually they moved it out into a userland server. Apple nowadays has a lot of filing systems run in userspace but not the core APFS that's used for most stuff, which is still in-kernel. Android moved a lot of stuff out of the kernel with time too. It has to be taken on a case by case basis.
Nintendo's 3DS OS and Switch 1+2 OS are bespoke and strictly microkernel-based (with the exception of DMA-330 CoreLink DMA handling on 3DS if you want to count is as such), and these have been deployed on hundreds of millions of commercially-sold devices.
Can you explain why TTY-PTY functionality hasn't been moved from the Linux kernel to userspace? Plan 9 did so in the 1990s or earlier (i.e., when Plan 9 was created, they initially put the functionality in userspace and left it there.)
I don't understand that, and I also don't understand why users who enjoy text-only interaction with computers are still relying on very old designs incorporating
things like "line discipline", ANSI control sequences and TERMINFO databases. A large chunk of cruft was introduced for performance reasons in the 1970s and even the 1960s, but the performance demands of writing a grid of text to a screen are very easily handled by modern hardware, and I don't understand why the cruft hasn't been replaced with something simpler.
In other words, why do users who enjoy text-only interaction with computers still emulate hardware (namely, dedicated terminals) designed in the 1960s and 1970s that mostly just displays a rectangular grid of monospaced text and consequently would be easy to implement afresh using modern techniques?
There a bunch of complexity in every terminal emulator for example for doing cursor-addressing. Network speeds are fast enough these days (and RAM is cheap enough) that cursor-addressing is unnecessary: every update can just re-send the entire grid of text to be shown to the user.
Also, I think the protocol used in communication between the terminal and the computer is stateful for no reason that remains valid nowadays.
The usual reason for all of this is that programmer time is expensive (even if you're a volunteer, you have limited hours available), and not many people want to volunteer to wade through tons of legacy tech debt. That's especially true when the outcome will be an OS that behaves identically to before. A lot of stuff stays in the kernel because it's just hard to move it out.
Bear in mind, moving stuff out of the kernel is only really worth it if you can come up with a reasonable specification for how to solve a bunch of new problems. If you don't solve them it's easy to screw up and end up with a slower system yet no benefit.
Consider what happens if you are overenthusiastic and try to move your core filesystem into userspace. What does the OS do if your filesystem process segfaults? Probably it can't do anything at that point beyond block everything and try to restart it? But every process then lost its connection to the FS server and so all the file handles are suddenly invalidated, meaning every process crashes. You might as well just panic and reboot, so, it might as well stay in the kernel. And what about security? GNU Hurd jumped on the microkernel bandwagon but ended up opening up security vulnerabilities "by design" because they didn't think it through deeply enough (in fairness, these issues are subtle). Having stuff be in the kernel simplifies your architecture tremendously and can avoid bugs as well as create them. People like to claim microkernels are inherently more secure but it's not the case unless you are very careful. So it's good to start monolithic and spin stuff out only when you're ready for the complexity that comes with that.
Linux also has the unusual issue that the kernel and userspace are developed independently, which is an obvious problem if you want to move functionality between the two. Windows and macOS can make assumptions about userspace that Linux doesn't.
If you want to improve terminals then the wrong place to start is fiddling with moving code between kernel and user space. The right place to start is with a brand new protocol that encodes what you like about text-only interaction and then try to get apps to adopt it or bridge old apps with libc shims etc.
>Consider what happens if you are overenthusiastic and try to move your core filesystem into userspace. What does the OS do if your filesystem process segfaults? Probably it can't do anything at that point beyond block everything and try to restart it? But every process then lost its connection to the FS server and so all the file handles are suddenly invalidated, meaning every process crashes. You might as well just panic and reboot, so, it might as well stay in the kernel.
I mean, it's not necessarily true that if a filesystem process crashes, every other process crashes. Depending on the design, each FS process may serve requests for each mountpoint, or for each FS type. That already is a huge boon to stability, especially if you're using experimental FSs. On top of that, I think the broken connection could be salvageable by the server storing handle metadata in the kernel and retrieving it when the kernel revives the process. It's hardly an insurmountable problem.
Some simpler CPU boards for embedded systems have no onboard graphics, they just have a serial port, so you have to use a terminal or terminal emulator to talk to them.
Thanks, it's great to see all the issues you raise.
On the other hand, it seems about as hard as I was imagining. I take for granted that it has to be a new language -- you obviously can't add it on top of Python, for example. And obviously it isn't compatible with things like global monkeypatching.
But if a language's built-in functions are built around the idea from the ground up, it seems entirely feasible. Particularly if you make the limits entirely around permissions around data communication -- with disk, sockets, APIs, hardware like webcams and microphones, and "god" permissions like shell or exec commands -- and not about trying to merely constrain resource usage around things like CPU, memory, etc.
If a package is blowing up your memory or CPU, you'll catch it quickly and usually the worst it can do is make your service unavailable. The risk to focus on should be exclusively data access+exfiltration and external data modification, as far as I can tell. A package shouldn't be able to wipe your user folder or post program data to a URL at all unless you give it permission. Which means no filesystem or network calls, no shell access, no linked programs in other languages, etc.
tbh none of that sounds particularly bad, nor do I think capabilities are necessary (but obviously useful).
we could literally just take Go and categorize on "imports risky package" and we'd have a better situation than we have now, and it would encourage library design that isolates those risky accesses so people don't worry about them being used. even that much should have been table stakes over a decade ago.
and like:
>No language has such an object or such interfaces in its standard library, and in fact “god objects” are viewed as violating good object oriented design.
sure they do. that's dependency injection, and you'd probably delegate it to a dependency injector (your god object) that resolves permissions. plus go already has an object for it that's passed almost everywhere: context.
perfect isn't necessary. what we have now very nearly everywhere is the most extreme example of "yolo", almost anything would be an improvement.
Yes, dependency injection can help although injectors don't have any understanding of whether an object really needs a dependency. But that's not a god object in the sense it's normally meant. For one, it's injecting different objects :)
to be clear, I mean that the DI container/whatever is "the god object" - it holds essentially every dependency and every piece of your own code, knows how to construct every single one, and knows what everything needs. it's the biggest and most complicatedly-intertwined thing in pretty much any application, and it works so well that people forget it exists or how it works, and carrying permission-objects through that on a library level would be literally trivial because all of them already do everything needed.
hence: doesn't sound too bad
"truly needs": currently, yes. but that seems like a fairly easy thing to address with library packaging systems and a language that supports that. static analysis and language design to support it can cover a lot (e.g. go is limited enough that you can handle some just from scanning imports), and "you can ask for something you don't use, it just means people are less likely to use your library" for the exceptions is hardly a problem compared to our current "you already have every permission and nobody knows it".
Thanks, this was a good overview of some of the challenges involved with designing a capability language.
I think I need to read up more on how to deal with (avoiding) changes to your public APIs when doing dependency injection, because that seems like basically what you're doing in a capability-based module system. I feel like there has to be some way to make such a system more ergonomic and make the common case of e.g. "I just want to give this thing the ability to make any HTTP request" easy, while still allowing for flexibility if you want to lock that down more.
In Java DI you can add dependencies without changing your public API using field injection. But really there needs to be a language with integrated DI. A lot of the pain of using DI comes from the way it's been strapped on the side.
Yes, but that was with a very ambitious sandbox that included full GUI access. Sandboxing a pure data transformation utility like something that strips ANSI escape codes would have been much easier for it.
He says that in the article; he had to work on a "compiler" project that was much harder than it should have been because of protobuf's design choices.
Yeah, I saw that. I took that as something that happened in the past, though. Certainly colored a lot of the thinking, but feels like something more immediate had to have happened. :D
People under-estimate how much code gets written in these languages just because decades ago they were chosen as the default language of the project and people are resistant to going full polyglot. Then everything gets written that way including cold paths, utilities, features that are hardly ever used, UI code that's bottlenecked by the network...
Chrome is a bad example. It uses a tracing GC in its most performance sensitive parts explicitly to reduce the number of memory safety bugs (it's called Oilpan). And much of the rest is written in C++ simply because that's the language Chrome standardized on, they are comfortable relying on kernel sandboxes and IPC rather than switching to a more secure language.
> Rust can be used anywhere in the Chromium repository (not just //third_party) subject to current interop capabilities, however it is currently subject to a internal approval and FYI process. Googlers can view go/chrome-rust for details. New usages of Rust are documented at rust-fyi@chromium.org.
It is true that two years ago it was only third party, but it's been growing ever since.
The only thing I intimated about Chrome is that if it got 2x slower, many users would in fact care. I have no doubt that they very well might not write it in C++ if they started today (well, if they decided not to start with a fork of the WebKit HTML engine). I’m not sure what Oilpan has to do with anything I said - I suspect that it would do memory operations too opaque for Fil-C’s model and V8 certainly would but I don’t see how that makes it a bad example of performance-sensitive C++ software.
Adaptive planning also applies to adapting the degree of query parallelism and automatic banning of query plans that take too much time.
Commercial databases have much bigger teams than Postgres does and have a lot more optimizations as a consequence. This includes optimizations that aren't in scope for an open source DB to begin with. For example, Oracle uses the Apple strategy for decades. A lot of Oracle DBs are sold as hardware not software, shipped as a full cluster rack or set of racks to the customer. The gear uses a dedicated inter-node 100Gb/sec ethernet within the cluster to give horizontal scalability of joins and other complex read/write loads. Queries and results go in/out over a separate frontend network. There are customizations up and down the stack from kernel, firmware, device drivers, database nodes and the client drivers. Nodes directly read/write each other's memory using RDMA to fetch the latest version of rows from each other or inform each other about new transactions.
It's true that they do this sort of thing for political reasons, but it sounds like the original NYT report wasn't meant to be merely a paraphrase of a specific UN report? In which case, it would be legitimate to cite other sources and report that they disagree?
No it was a paraphrase of the report and cited no sources that disagreed. It simply sneakily misrepresented the contents and buried the links to the actual report.
There's a simpler explanation: they are comparing LLM performance to that of regular humans, not perfection.
Where do you think LLMs learned this behavior from? Go spend time in the academic literature outside of computer science and you will find an endless sea of material with BS citations that don't substantiate the claim being made, entirely made up claims with no evidence, citations of retracted papers, nonsensical numbers etc. And that's when papers take months to write and have numerous coauthors, peer reviewers and editors involved (theoretically).
Now read some newspapers or magazines and it's the same except the citations are gone.
If an LLM can meet that same level of performance in a few seconds, it's objectively impressive unless you compare to a theoretical ideal.
They didn't cited papers directly even before the web. It's not a bounce or engagement issue.
Journalists don't make it easy for you to access primary sources because of a mentality and culture issue. They see themselves as gatekeepers of information and convince themselves that readers can't handle the raw material. From their perspective, making it easy to read primary sources is pure downside:
• Most readers don't care/have time.
• Of the tiny number who do, the chances of them finding a mistake in your reporting or in the primary source is high.
• It makes it easier to mis-represent the source to bolster the story.
Eliminating links to sources is pure win: people care a lot about mistakes but not about finding them, so raising the bar for the few who do is ideal.
That’s not how it works at large news orgs. Journalists will enter their articles in a CMS. From there it will get put into a workflow and seen by one or several editors who will edit things for clarity, grammar, style, etc. Links will get added at some point by an editor or automated system. There is no cabal of journalists scheming to keep links out of articles because “culture”.
If there were a culture of always including the original source, or journalists massively advocating to include the original source, then surely the CMS would cater to it. I think it's safe to draw the conclusion that most journalists don't care about it.
There’s a lot of rightly deserved criticism of the media but the OP describing journalists as conspiring to keep links out in the fear of being fact checked by readers is simply false and indicative of not having any experience at a large news organization.
It's not historically done. Printed newspapers obviously didn't have links and neither did televised news. Even when the news media started publishing online, it's not like the courts were quick to post the decisions online.
And there's also the idea that you should be able to at least somewhat trust the people reporting the news so they don't have to provide all of their references. --You can certainly argue that not all reporters can or should be trusted anymore, but convincing all journalists to change how they work because of bad ones is always going to be hard.
There is also the added pressure that some organizations quietly pile on editors to keep people from clicking out to third parties at all, where their attention may wander away. Unless of course that third party is an ad destination.
Reputable news organizations are more robust against such pressures, but plenty of people get their news from (in some cases self-described) entertainment sites masquerading as news sites.
Is it because journalists think of their special talent as talking to people to get information (which is a scarce and priviliged resource), versus reading and summarizing things that we all have access to?
So they rarely are forced to do anything but state the name of who they interviewed, and that's it. And puts them in the habit of not acknowledging what they read, as a source?
It seems more malicious or intentional than only trying to gate keep. More often then not when I dig deeper into a news article referencing a scientific paper or study, the details they pull out are very much out of context and don't tell the same story as the research.
I have to assume the journalist writing such an article knows that they are misrepresenting the research to make a broader point they want to make.
There's not much to say about it because there's never been an actual disagreement in philosophy. Every OS designer knows it's better for stability and development velocity to have code run in userspace and they always did. The word microkernel came from academia, a place where you can get papers published by finding an idea, giving it a name and then taking it to an extreme. So most microkernels trace their lineage back to Mach or similar, but the core ideas of using "servers" linked by some decent RPC system can be found in most every OS. It's only a question of how far you push the concept.
As hardware got faster, one of the ways OS designers used it was to move code out of the kernel. In the 90s Microsoft obtained competitive advantage by having the GUI system run in the kernel, eventually they moved it out into a userland server. Apple nowadays has a lot of filing systems run in userspace but not the core APFS that's used for most stuff, which is still in-kernel. Android moved a lot of stuff out of the kernel with time too. It has to be taken on a case by case basis.
reply