>"It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions."
An excellent point!
>"Yes pipelines are a big deal and the runtime is solid now so they can be enhanced."
Maybe synchronous (aka, guaranteed linear) and asynchronous (no guarantees) are better ways to look at it...
For example, a simple shell command like 'echo' could be run either synchronously or asynchronously, but if we're assigning the result of it to some variable, it makes sense to run it synchronously, to get that result into the variable before proceeding to the next line of the program...
But, if we're running a disk defragmenting tool, or other process that we don't know when it's going to complete, then it makes sense to run it asynchronously.
Shell commands that are chained together via pipelines may run asynchronously, but the shell may wait for everything to complete to scoop up the output; so you've sort of have the entire line that was sent to the shell running synchronously (if you're assigning the result to a variable), even though parts of it are running asynchronously (with respect to one another)...
And of course, via ampersand, you could have the entire line run asynchronous to your program...
See, I'd almost say that one of the problems is that shell scripting languages almost have no knowledge (and correct me if I am wrong, I might be!) about the pipes they create inside of commands -- an ideal shell language would be able to know about these and hook these for various purposes, and specify what would be ran synchronously and asynchronously...
Or, another weird corner case -- what if you have a shell script that launches other shell scripts asynchronously (in subshells) and they launch other shell scripts also asynchronously -- in deeper level subshells?
How would your language/shell track/handle that case?
How do you communicate/track/determine when/where to assign results to things?
It's sort of like the language/shell -- must be able to :
A) Explicitly say what is asynchronous and what is synchronous...
B) For asynchronous things, be able to communicate with them, track their status, get results from them when they complete, etc., etc.
Now, of all of the programming languages in existence, I think Go is a good candidate for understanding synchronous vs. non-synchronous -- but you don't exactly get an "easily scriptable" beginner-friendly shell-scripting language with Go...
Python seems to be the right balance between "ease of use" and "gives you control".
So I'm agreed with all that you've said about Python, and your use of Python in code...
Yes being able to store the FDs for pipes in variables is something that would make it more explicit and flexible. (It could also introduce deadlocks though)
All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes...
That's sort of a good thing and a bad thing at the same time...
It's great for such things as binary files, binary streams, anything binary in nature...
But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you.
I also like your point about FD's for pipes... that's a good idea too.
Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc.
Anyway, some fascinating stuff!
>"Although I think richer pipelines make sense, structured data will probably come first:
An excellent point!
>"Yes pipelines are a big deal and the runtime is solid now so they can be enhanced."
Maybe synchronous (aka, guaranteed linear) and asynchronous (no guarantees) are better ways to look at it...
For example, a simple shell command like 'echo' could be run either synchronously or asynchronously, but if we're assigning the result of it to some variable, it makes sense to run it synchronously, to get that result into the variable before proceeding to the next line of the program...
But, if we're running a disk defragmenting tool, or other process that we don't know when it's going to complete, then it makes sense to run it asynchronously.
Shell commands that are chained together via pipelines may run asynchronously, but the shell may wait for everything to complete to scoop up the output; so you've sort of have the entire line that was sent to the shell running synchronously (if you're assigning the result to a variable), even though parts of it are running asynchronously (with respect to one another)...
And of course, via ampersand, you could have the entire line run asynchronous to your program...
See, I'd almost say that one of the problems is that shell scripting languages almost have no knowledge (and correct me if I am wrong, I might be!) about the pipes they create inside of commands -- an ideal shell language would be able to know about these and hook these for various purposes, and specify what would be ran synchronously and asynchronously...
Or, another weird corner case -- what if you have a shell script that launches other shell scripts asynchronously (in subshells) and they launch other shell scripts also asynchronously -- in deeper level subshells?
How would your language/shell track/handle that case?
How do you communicate/track/determine when/where to assign results to things?
It's sort of like the language/shell -- must be able to :
A) Explicitly say what is asynchronous and what is synchronous...
B) For asynchronous things, be able to communicate with them, track their status, get results from them when they complete, etc., etc.
Now, of all of the programming languages in existence, I think Go is a good candidate for understanding synchronous vs. non-synchronous -- but you don't exactly get an "easily scriptable" beginner-friendly shell-scripting language with Go...
Python seems to be the right balance between "ease of use" and "gives you control".
So I'm agreed with all that you've said about Python, and your use of Python in code...