I don't think it is essential that there be a clear boundary between an OS and programs.
For example, you can have a Smalltalk-based OS, written in Smalltalk. The OS components are simply Smalltalk classes and methods. Your programs/utilities/applications would also be Smalltalk classes and methods. There needn't be any clear boundary between the classes/methods that belong to the OS and those that belong to programs/applications/utilities.
If you look at library operating systems, the operating system is just a library which you link into your application. Such an operating system can only run one program, and you have to rebuild the operating system whenever you want to change the single program it runs. Still, I think library operating systems count as operating systems. This Tetris implementation isn't using a library operating system, but you could refactor the code into two parts – the Tetris game, and generic OS services, and the later would constitute a library operating system. If you look at its source files, only about seven of them (main.c, music.[ch], sound.[ch], speaker.[ch]) actually implement the game. The other 23 source files are generic OS services.
So I'd say that, even if this isn't an operating system as a whole, it contains an OS within itself.
I like to define an OS as something that can execute arbitrary binaries. So if your hypothetical Smalltalk OS has a defined executable format that non-Smalltalk programs can compile to, I’d call it an OS.
Note that this also makes for example the uppermost part of the JVM an OS by my definition. I’d argue it’s not a completely incorrect characterization.
I think that definition is a bit too broad and too narrow to be useful.
For too narrow: you might have a security focused OS that includes everything like a TCP/IP stack etc, but perhaps doesn't allow arbitrary binaries by construction. (Eg it might only allow binaries that come with a proof of innocence, or some other restriction that's harsh enough to lose the 'arbitrary' rating.)
Obviously, "capable of running arbitrary binaries, but with a check inside exec() to allow only specific binaries run" counts as "capable of running arbitrary binaries".
You've presented a conscientious criterion for what can be classified as an operating system. I think it's clever, I'm not sure I agree though. It's very broad: By this same definition any processor is an operating system, since it will execute arbitrary binary data. Your example of the JVM being an operating system calls to mind the above: It is a virtual machine, it is designed to execute arbitrary bytecode in a similar manner to a processor.
You may have found a loophole in my specification :)
I’d like to amend it to specify that an OS refers to software “that can run...”. (And this is why we do design reviews!).
In your example, a real or virtual machine’s load/setregs interface is an operating system. One that can only run a single program at a time in the case of the physical processor’s bootstrap, but you’d be surprised how similar that is to early DOS.
A more hand-waving version is “the lowest level interface over the machine that can run...”. So a processor and a ROM grid can run arbitrary programs, but the lowest common denominator entry point is the ROM flashing spec! A very slow scheduler indeed.
What do you mean by "arbitrary binary"? Surely, you don't mean that absolutely any binary blob should be a meaningful executable. Because that would mean that my computer doesn't have an operating system because it's only capable of running binaries built for Ubuntu on amd64 -- if your binaries targeting BSD on ARM can be viewed as "arbitrary," then surely they shouldn't be expected to run on my pseudo-operating system.
No, every operating system that I know of makes some assumptions about the language that programs are written in. On my amd64/Ubuntu system, the language is largely specified by the processor (and, yes, has a binary format) and the programs are allowed to assume the presence of various devices, formats of various system calls, etc. Other languages are supported through a complicated system of compilers, interpreters and assemblers.
Or, coming at a different tack: does the existence of a C->Smalltalk transpiler turn the Smalltalk-based pseudo-OS into a real OS? What if the pseudo-OS doesn't ship the transpiler, and you need to run that yourself in user-space?
I don't agree that something has to be capable of running "arbitrary binaries" to count as an operating system. If I have a Smalltalk-based OS which is capable of running arbitrary Smalltalk code, but doesn't have any support for running non-Smalltalk binaries (native code or programs written in languages other than Smalltalk), I'd still consider it to be an operating system.
The “arbitrary” part is a force of how rather than what programs are run.
Is your Smalltalk code running by compiling it into the OS? Is it loaded as a library? Can I write a C backend to output an indistinguishable library? In the latter cases, it is still capable of running arbitrary code - you may just be choosing not to.
You can still write code in a different language like C that will compile into Smalltalk VM bytecode. You will never be able to run any arbitrary code, just the one compiled for your machine. And in this case, the machine is virtual (but does not need to be). Symbolics Lisp machines were able to run Pascal as well, for example.
For example, you can have a Smalltalk-based OS, written in Smalltalk. The OS components are simply Smalltalk classes and methods. Your programs/utilities/applications would also be Smalltalk classes and methods. There needn't be any clear boundary between the classes/methods that belong to the OS and those that belong to programs/applications/utilities.
If you look at library operating systems, the operating system is just a library which you link into your application. Such an operating system can only run one program, and you have to rebuild the operating system whenever you want to change the single program it runs. Still, I think library operating systems count as operating systems. This Tetris implementation isn't using a library operating system, but you could refactor the code into two parts – the Tetris game, and generic OS services, and the later would constitute a library operating system. If you look at its source files, only about seven of them (main.c, music.[ch], sound.[ch], speaker.[ch]) actually implement the game. The other 23 source files are generic OS services. So I'd say that, even if this isn't an operating system as a whole, it contains an OS within itself.