I recently ported WebKit's libpas memory allocator[1] to Windows, which used pthreads on the Linux and Darwin ports. Depending on what pthreads features you're using it's not that much code to shim to Windows APIs. It's around ~200 LOC[2] for WebKit's usage, which a lot smaller than pthread-win32.
This is something you also need to do for other Win32 APIs, e.g. file write access may be temporarily blocked by anti-virus programs or whatever and not handling that makes unhappy users.
I'm a big fan of pigz, I discovered it 6 years ago when I had some massive files I needed to zip and and 48 core server I was underutilizing. It was very satisfying to open htop and watch all the cores max out.
-Premake supports Visual Studio 2008 and 2010 (and 2012 supports 2010 project files via conversion).
+Premake supports latest Visual Studio 2018 and 2022 project files via conversion).
Perhaps it's worth it adding this as a note at the top of the post, maybe mentioning alternatives, such as an Actually Portable™ build of `pigz`[1] or just a windows build of zstd[2].
I don't think the port itself is very old. The latest version of original pigz seems to have been released in 2023 [1], and the port seems to be of pigz from around that time[2]
I'm not sure how willing I'd be to trust a pthread library fork from a single no-name github person. The mingw-w64 project provides libwinpthread, which you can download as source from their sourceforge, or as a binary+headers from a well-known repository like msys2.
> Porting pthreads code to Windows would be a nightmare.
Porting one application using pthreads to use the Win32 API directly is however a lot more reasonable and provides you more opportunity to deal with impedance mismatches than a full API shim has. Same goes for dirent and other things as well as for the reverse direction. Some slightly higher level abstraction for the thnings your program actually needs is usually a better solution for cross-platform applications than using one OS API and emulating it on other systems.
I'd rather everyone use CMake than have to deal with yet another build system. Wouldn't be so bad if build systems could at least agree on the user interface and package registry format.
depends on the current load. i've worked places where we would create nightly postgres dumps via pg_dumpall, then pipe through pigz to compress. it's great if you run it when load is otherwise low and you want to squeeze every bit of performance out of the box during that quiet window.
this predates the maturation of pg_dump/pg_restore concurrency features :)
Not to over state it, embedding the parallelism into the application drives to the logic "the application is where we know we can do it" but embedding the parallelism into a discrete lower layer and using pipes drives to "this is a generic UNIX model of how to process data"
The thing with "and pipe to <thing>" is that you then reduce to a serial buffer delay decoding the pipe input. I do this, because often its both logically simple and the component of serial->parallel delay deblocking on a pipe is low.
Which is where xargs and the prefork model comes in, because instead you segment/shard the process, and either don't have a re-unification burden or its a simple serialise over the outputs.
When I know I can shard, and I don't know how to tell the appication to be parallel, this is my path out.
[1] https://github.com/WebKit/WebKit/pull/41945 [2] https://github.com/WebKit/WebKit/blob/main/Source/bmalloc/li...