First off, any person who would write either a language or a shell, now or in the future, should read your page, and in fact, should read everything about your Oil shell/language...
It's a laudable effort!
This page is going to my HN favorites, for future review.
What follows next are my thoughts about selected excerpts of text on the page (please don't interpret as criticism, that's not the intent):
>"You can think of a Unix shell in two ways:
1. As a text-based user interface. You communicate with the operating system by typing commands.
2. As a language. It has variables, functions, and loops. Shell programs are text files that start with #!/bin/sh."
[...]
>"Are you reinventing Perl?
It's true that Perl is closer to shell than Python and Ruby are. For example, the perl -pie idiom can take the place of awk and sed. However, Perl isn't an acceptable shell either:
o It doesn't have true pipelines (of processes). I believe you have to "shell out" for this.
o It doesn't have a concept of file descriptor state. How do you write the equivalent of my_shell_func 2> err.txt, where my_shell_func can invoke both functions and external commands?
o Perl has a function called grep(), but the real grep is better for many problems."
PDS: This touches upon a historic problem, which is basically that either:
A) Shell designers work their way "up" to a programming language, in which case some elegant programming language features are either not thought about, or not thought deeply enough about, and thus either not implmented, or implemented poorly / "non-orthagonally", AKA a "Kludge".
Or:
B) Language designers work their way "down" to supporting shell commands / a shell subsystem -- in which case some elegant shell features are either not thought about, or not thought deeply enough about, and thus either not implmented, or implemented poorly / "non-orthagonally", AKA a "Kludge".
Also, languages are typically too heavy in what they require typed for shell-scripters, and shells are typically too light in what they allow (and error checking/debugging features) for programmers writing large programs...
If someone were to design the most ideal programming language which also can be used for shell scripting, they'd have to look at things from both perspectives, and they'd have to create a language/shell -- completely balanced "down the middle" between these two paradigms...
Perhaps the Oil shell/programming language -- is, or will become that middle line...
To that extent, I wish you a lot of luck!
If I were going to go down this path, I'd look at "first principles" (Elon Musk):
The greatest differentiator between shell commands and lines of code in most programming languages is the point you alluded to in your comparison with Perl:
"o It doesn't have true pipelines (of processes). I believe you have to "shell out" for this."
In other words, in most programming languages (unless you are using threads), you are guaranteed linear execution.
In a Unix shell -- because you can run multiple commands that may take varying amounts of time to complete, this guarantee is no longer present...
That would be the fundamental thing to keep in mind when writing this future language/shell...
(A good program to test when writing this language would be a Web-server -- where the base server is a single script, and then when it detects a connection, passes this over to either a single separate command-line invoked Unix program or script comprising multiple such Unix programs linked with pipes, but the central "server" script has to handle a whole lot of simultaneous I/O from multiple such separate programs (it centralizes database access!), and do this in a simple way that script programmers would be comfortable with, while guaranteeing the correct data flows...)
(author here) Thanks for the comment! I do think you hit on something interesting, and it partly explains why the project is so big :)
To me, shell and Python/JS feel kind of similar, and that was sort of the thesis at the start of the project.
But if you just incrementally add features to Bourne shell, you basically get Korn shell, which is where bash got all the "bashisms" that people don't like.
This paper was surprising to me; the proprietary AT&T ksh was used for GUIs and so forth, AND it was embeddable like Lua:
[ksh] has the capabilities of perl and tcl, yet it is backward compatible with the Bourne shell. Applications of up to 25,000 lines have been written in ksh and are in production use.
Much of the impetus for ksh–93 was
wksh, which allows graphical user interfaces to be written in ksh.
It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions.
So I would say that ksh "failed" because it took a lot of shortcuts. Perl, Python, Ruby, JS, and PHP won. So with Oil I also am upgrading shell, but it was a lot more work since I wanted to make it like the latter languages. It wasn't obvious at the outset how different these things are!
So in a sense Oil is upgrading shell to be more like the languages that won, which I still think is a good idea. It's weird that every computer boots to into a programming language REPL, but you're discouraged from learning it because it sucks. There's no reason that language shouldn't be a good one.
>"It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions."
An excellent point!
>"Yes pipelines are a big deal and the runtime is solid now so they can be enhanced."
Maybe synchronous (aka, guaranteed linear) and asynchronous (no guarantees) are better ways to look at it...
For example, a simple shell command like 'echo' could be run either synchronously or asynchronously, but if we're assigning the result of it to some variable, it makes sense to run it synchronously, to get that result into the variable before proceeding to the next line of the program...
But, if we're running a disk defragmenting tool, or other process that we don't know when it's going to complete, then it makes sense to run it asynchronously.
Shell commands that are chained together via pipelines may run asynchronously, but the shell may wait for everything to complete to scoop up the output; so you've sort of have the entire line that was sent to the shell running synchronously (if you're assigning the result to a variable), even though parts of it are running asynchronously (with respect to one another)...
And of course, via ampersand, you could have the entire line run asynchronous to your program...
See, I'd almost say that one of the problems is that shell scripting languages almost have no knowledge (and correct me if I am wrong, I might be!) about the pipes they create inside of commands -- an ideal shell language would be able to know about these and hook these for various purposes, and specify what would be ran synchronously and asynchronously...
Or, another weird corner case -- what if you have a shell script that launches other shell scripts asynchronously (in subshells) and they launch other shell scripts also asynchronously -- in deeper level subshells?
How would your language/shell track/handle that case?
How do you communicate/track/determine when/where to assign results to things?
It's sort of like the language/shell -- must be able to :
A) Explicitly say what is asynchronous and what is synchronous...
B) For asynchronous things, be able to communicate with them, track their status, get results from them when they complete, etc., etc.
Now, of all of the programming languages in existence, I think Go is a good candidate for understanding synchronous vs. non-synchronous -- but you don't exactly get an "easily scriptable" beginner-friendly shell-scripting language with Go...
Python seems to be the right balance between "ease of use" and "gives you control".
So I'm agreed with all that you've said about Python, and your use of Python in code...
Yes being able to store the FDs for pipes in variables is something that would make it more explicit and flexible. (It could also introduce deadlocks though)
All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes...
That's sort of a good thing and a bad thing at the same time...
It's great for such things as binary files, binary streams, anything binary in nature...
But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you.
I also like your point about FD's for pipes... that's a good idea too.
Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc.
Anyway, some fascinating stuff!
>"Although I think richer pipelines make sense, structured data will probably come first:
First off, any person who would write either a language or a shell, now or in the future, should read your page, and in fact, should read everything about your Oil shell/language...
It's a laudable effort!
This page is going to my HN favorites, for future review.
What follows next are my thoughts about selected excerpts of text on the page (please don't interpret as criticism, that's not the intent):
>"You can think of a Unix shell in two ways:
1. As a text-based user interface. You communicate with the operating system by typing commands.
2. As a language. It has variables, functions, and loops. Shell programs are text files that start with #!/bin/sh."
[...]
>"Are you reinventing Perl?
It's true that Perl is closer to shell than Python and Ruby are. For example, the perl -pie idiom can take the place of awk and sed. However, Perl isn't an acceptable shell either:
o It doesn't have true pipelines (of processes). I believe you have to "shell out" for this.
o It doesn't have a concept of file descriptor state. How do you write the equivalent of my_shell_func 2> err.txt, where my_shell_func can invoke both functions and external commands?
o Perl has a function called grep(), but the real grep is better for many problems."
PDS: This touches upon a historic problem, which is basically that either:
A) Shell designers work their way "up" to a programming language, in which case some elegant programming language features are either not thought about, or not thought deeply enough about, and thus either not implmented, or implemented poorly / "non-orthagonally", AKA a "Kludge".
Or:
B) Language designers work their way "down" to supporting shell commands / a shell subsystem -- in which case some elegant shell features are either not thought about, or not thought deeply enough about, and thus either not implmented, or implemented poorly / "non-orthagonally", AKA a "Kludge".
Also, languages are typically too heavy in what they require typed for shell-scripters, and shells are typically too light in what they allow (and error checking/debugging features) for programmers writing large programs...
If someone were to design the most ideal programming language which also can be used for shell scripting, they'd have to look at things from both perspectives, and they'd have to create a language/shell -- completely balanced "down the middle" between these two paradigms...
Perhaps the Oil shell/programming language -- is, or will become that middle line...
To that extent, I wish you a lot of luck!
If I were going to go down this path, I'd look at "first principles" (Elon Musk):
The greatest differentiator between shell commands and lines of code in most programming languages is the point you alluded to in your comparison with Perl:
"o It doesn't have true pipelines (of processes). I believe you have to "shell out" for this."
In other words, in most programming languages (unless you are using threads), you are guaranteed linear execution.
In a Unix shell -- because you can run multiple commands that may take varying amounts of time to complete, this guarantee is no longer present...
That would be the fundamental thing to keep in mind when writing this future language/shell...
(A good program to test when writing this language would be a Web-server -- where the base server is a single script, and then when it detects a connection, passes this over to either a single separate command-line invoked Unix program or script comprising multiple such Unix programs linked with pipes, but the central "server" script has to handle a whole lot of simultaneous I/O from multiple such separate programs (it centralizes database access!), and do this in a simple way that script programmers would be comfortable with, while guaranteeing the correct data flows...)
Anyway, wishing you luck in this endeavor!