I am planning to use this for my homemade system-wide update scripts -- that updates via the OS package manager, Golang tools, Rust tools, OCaml tools, NPM-managed tools, editor configs (NeoVim headless updating of plugins as well), and others.
Seems like it has everything I need, the most crucial ones being separate queues / worker pools (f.ex. I only want the tasks that might involve source compilation to be executed one after another on a separate queue, never in parallel).
Anybody knows a good alternative to `multitail`? Still haven't found any other tool that can track the output of several commands at the same time on a screen that's automatically split -- and subsequently the finished tasks get removed from view until only the last one remains, taking the entire screen, and then it says "Done, press Q to quit".
What exactly is the is doing that you can’t do with pure bash? You can set up chains of commands in serial or in parallel, use job arrays, establish dependencies or other run conditions, hop into a backgrounded run, etc.
You can probably achieve a good subset of its functionality in bash, it's just a nicer interface with a lot of configurability and several convenience features.
Would you be willing to write a proper guide on how to do all of these things in bash? It would be great to have such a guide inside the Pueue wiki and link to it. It'll help people to make a more informed decision on whether they need this tool or not.
Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally.
I was running a lot of small tasks. With 3000 queued tasks my state file was around 25 MB.
zstd compressed that down to 5% of the size. I have the code still if you want to look at it but it was just a quick experiment so I didn't add any tests. I did add it to the config, disabled by default, though.
"pueue follow <task_id>" lets you see stdout or stderror of the specified task.
If one enqueues a single chain of tasks (no parallel tasks), is there a way to monitor stdout or stderror for the chain, without having to issue the follow command for each task at the time the task starts to run? This would provide better observability of what is running, as in a shell script with the tasks sequentially listed.
At first I thought it would just be a one-off tool I used for one of my projects, not until I discovered later that it has everything I need and became my daily driver ever since.
I've wanted to write something like this for years! For me it's always long running rsync commands I want to chain.
I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I'll be checking this out.
Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.
If this ability is important to you then you can break your pipeline into individual files that aren’t going to be read until they are executed, giving you time to edit.
Fair point, but to be honest, at this point it's just easier to do a:
pueue add 'rsync somestuff host:location'
And if I notice any problems, I just do a `pueue edit $id` and I'm good to go.
It's just a lot more convenient than manually building pipelines with files that'll be executed.
It would be something different if this was about recurrent tasks that needed to be done, though. But for one-off stuff, your approach seems a bit cumbersome.
Yes or I could use the bash `&&` but the issue is I need to know all the commands at the start. I want to be able to come back an hour later and easily add a command to the end of the chain.
Perhaps by sending SIGSTOP and SIGCONT, much like hitting Ctrl+Z on the console and later running bg <pid> or fg <pid>.
Note that this is not the same as Ctrl+S & Ctrl+Q on the console – that just pauses the output display not the process (though the process may subsequently pause if a buffer somewhere down the pipeline becomes full due to the terminal output pausing).
I was about to write that this has a different use case, because managing screen/tmux sessions is very manual. However the repo iself states
> Pueue is not designed to be a programmable (scriptable) task scheduler/executor. The focus of pueue lies on human interaction.
So I also don't really see its usecase and probably would opt for tmux instead. If you had many workers to run, but still do it manually, you would use pueue? I would be interested in such a scenario!
The concurrency controls are not present using the tmux method. Though if I am running enough tasks for this to be an issue then something odd is happening.
The queue being persisted and surviving crashes could be useful, and isn't the case with tmux unless you manually script it up. Though it could be painful if the crash leaves things remaining tasks depend upon in an odd state…
The task tree could be useful: scheduling tasks to start once other tasks are completed.
You can do this in pure bash I believe. E.g. when you launch a command you can capture its pid then wait for that to exit as you expect before running further jobs.
Well, if you're planning on running 10 commands sequentially that might take a few hours each, you would have to keep those windows open for quite a while.
This looks awesome and the README seems to undersell it. Would it be possible for you to put code examples all over the readme so we can see what you’re talking about without needing to dig into the codebase?
Seems like it has everything I need, the most crucial ones being separate queues / worker pools (f.ex. I only want the tasks that might involve source compilation to be executed one after another on a separate queue, never in parallel).
Anybody knows a good alternative to `multitail`? Still haven't found any other tool that can track the output of several commands at the same time on a screen that's automatically split -- and subsequently the finished tasks get removed from view until only the last one remains, taking the entire screen, and then it says "Done, press Q to quit".