Compatability with the C ABI I feel isn't a necesssity. On platforms built on C it is, but if you wanted to invent a new machine running a new system language with the ABI messes of C exonerated, you could. You would need to devise a translation layer in the OS executor to be able to interpret classic C ABI .so or .dll files for legacy applications and so you could compile C on these platforms, but that isn't a prohibitively hard challenge.
The need for C isn't so much restricted by the operating system, but by the hardware manufacturers. The simple fact is that C is the defacto choice of language for hardware vendors to ship a compiler for, and that doesn't look to change any time. It's a standard that can't be avoided, unless we can work cooperatively with enough major vendors to change this.
This is a myth and part of the vicious circle here. I'm not a C hater by any means but the reason hardware vendors have historically included it is because there are free compilers that are nearly C enough that they can include and they have to include something. C is just a portable assembly, you can almost write a simplistic compiler in pure yacc and more than a couple of these C compilers hardware vendors produced weren't much more than that. Software guys then wrote in C and the circle continues, intel has a great C compiler now and in fact all the best optimizers are from hardware vendors but that wasn't always the case. There was also a time (quite possibly it's still that time) when you'd buy some customed up MIPS or PowerPC chip and get an ancient version of gcc that they'd hacked to support their shit and forward porting to a modern GCC wasn't always a straight forward thing to do. They have to provide something and tools usually isn't their business.
Now so much is built on the C non-runtime that Go, Ada, Rust, anything with any runtime looks completely backwards to a lot of system folk. What's funny, you look at Linux and there is a fairly small hardware dependent directory of code where the register poking happens, it's really not that much code but its called from all the drivers and such, and that's the holdup. It's not like that stuff makes up 50% of the code or anything.
X86 provided efficient call/return protocol instructions including display management, specifically targeting Pascal/Modula style languages. Microsoft Windows 3.1 even mandated this calling protocol. However old style C didn't use this API because it doesn't work with varargs and C doesn't have nested functions.
Instead Intel observed real code and optimised their instruction set to speed it up, locking everyone into lame languages like C. Attempts by Intel to break from this, for example with x64, have failed (x64 has two stacks).
Similarly segmented architectures, which implement extension of the Harvard machine model, have been largely unsupported in favour of linear addressing von Neumann machines, primarily because that's what C and Unix use. Interesting to see Microsoft adapt their OS designs based on this trend.
To support more advanced features in a language often requires only a few very low level primitives. This is why Linux kernel doesn't need much assembler, to do things like stack swapping. Control exchange is one of those things completely missing from C.
I wasn't implying that there's C specific hardware, or that there's any technical restriction for shipping a C compiler with a chip. You're just reiterating my point about it being crowd effect.
Also, when talking about systems programming, we can't just limit our scope to linux - there's hundreds of other kernel projects, and embedded systems which don't have the luxury of a massive portable codebase already - they only have a basic C compiler, library, debugger.
Using a different language doesn't eliminate C, because chances are, the compiler you're using for that language has been written in C, because it's all the manufacturer supplied. Compiler vendors contribute to this cycle by expecting their compiler to run on any platform if they write it in C.
I was suggesting that if we want to liberate ourselves from C, we need the hardware vendors on board too - since they're where the cycle begins.
Of course, you don't need the vendor specific compiler if you have a cross compiler, and can build a different language compiler from another platform - however, if you look at what's available for doing that, there's nothing other than GCC - which is why new languages are unlikely to replace C as the defacto systems language, unless they can replicate that functionality.
Well, what sort of compiler optimizations you can make with C do have an effect on how people create microprocessors. For instance if you could be sure you could always disambiguate stack and heap accesses you could use two separate and somewhat memory pipes and get a big power or performance win, since there are a lot of N^2 structures involved in keeping everything coherent.
Also, most computers have a GPU which can't really be programmed in straight C.
Doesn't almost all systems programming take place on platforms built on C? If creating a new machine and new language, convincing the existing systems programmers to move is going to take something very compelling.
Historically, the "turtles all the way down" approach has not been very successful outside of niches. Lisp, Forth, Java have all been used for OS's that never seemed to go anywhere.
I'd much rather see three distinct languages, with a shared base syntax and style (ie, common formatting of comments, common scope delimiters, common operand behavior - something like C / Java / Javascript).
That way, you could have your low level language use pointers, nullables, inline assembly, goto and other "dangerous" things that are the most efficient ways to do stuff. You wouldn't want to have that language crippled like modern C++ though, you would still want modules, a well planned object oriented syntax, a better solution to name mangling, and better readability.
You could have an interpreted Java/C# esque applications language that would mostly be similar to modern Java/C#, maybe with auto, a better namespaces implementation (in C#, a class defined in two places with the same namespace and signature have situational behavior that gets really messy, for example). You would use this language for the application layer.
Scripting could just be an autoboxed dynamic language like Python. Maybe have a whitespace significant and insignificant dialect. I'd like to see dialects of such a language, where the context it runs from dictates available modules. Since it would be permissions based, it could function as a shell, web, and glue language depending on where you use it.
I do agree on this sentiment though. The wide base of languages in modern computing makes the whole thing a lot messier than it needs to be, and the lack of any full stack that really ties together nicely and is well thought out without the smudges would be really nice.
