I wish more were done to push back against this consolidation of power by these platforms.
We're like frogs that have been stoking the fuel of our own pot.
It used to be that DRM was considered to be in conflict with the browser, because it was not acting on behalf of the user. If you must have DRM, then it is on the platform to shoehorn it in through an external plugin, like Silverlight.
When Firefox adopted EME extensions, I knew it was the beginning of the end; they were rolling out the red carpet for DRM. If we make DRM a switch that can simply be thrown, then it will become the norm, not the exception. And there have been proposals for years to DRM fonts and other absurdities. If a company insists on using DRM, then they should have to shoulder the burden of doing something that a browser was never support to support.
The nightmare that we're racing toward is you will only be permitted to cache a trickle of video at a time and your TPM attestation hardware must include a token in every HTTP request. Your browser will just be a software cablebox.
They aren't happy about URLs either and would love to require that if you want to share a reference to something, you have to do it on their terms, like generating a url in their app with a hash that expires and limited in how many times it can be viewed. I'm sure influencers will still have the privilege of unlimited sharing.
They've been slowly rolling this infrastructure out for the last decade. These are not isolated inconveniences, these are coffin nails.
> The alternative is that people complain "netflix doesn't work on firefox", switch to chrome instead, which is even worse.
So what if users complain? How is it better for Firefox to do something bad just because Google is doing something bad?
Firefox is supposed to provide an alternative to what's out there. Firefox also didn't support some popular proprietary Internet Explorer features, and they never attempted to. For a time, much of MySpace didn't work as well in Firefox. But I'm glad that Firefox didn't cave, even if some users complained that they couldn't make the scrollbars neon green or make music autoplay.
Not letting Microsoft or Google dictate how they implement a web browser worked out really well for them.
Chasing proprietary platforms has ruined them.
At the time EME was adopted Firefox was much more popular, I think 20% back in 2012. Video platforms were using Silverlight for DRM. There's a good chance that EME would not have gotten off the ground if Mozilla didn't embrace it, or at least not as quickly.
Mozilla should have taken a stand and refused to support EME when they had the chance. They would be better off than they are now. And there's a good chance Netflix would not have thrown away a double digit percentage of subscribers.
Instead they embraced DRM and now they have nothing.
That's almost certainly their rationale, bit I'm not convinced it's sound. Firefox's market share is pretty dire anyway, and many people watch Netflix through phones and tv apps now, rather than their actual browser, I wouldn't be surprised if it wasn't an issue at all.
I have never thought of this and it's brilliant. I should definitely be able to whip up a relatively simple script for tagging arbitrary files with arbitrary tags. I was wondering how to achieve something like this for backup purposes and this should make it a breeze.
If you are iterating over a lot of files, a read while loop can be a major bottleneck. As long as you use the null options from find and pipe into xargs, you should be safe with any filename.
I've found it can reduce minutes down to seconds for large operations.
If you have to process a large number of files, you can let xargs minimize the number of times a program is run, instead of running it once per file.
Something like:
# Set the setgid bit for owner and group of all folders
find . -type d -print0 | xargs -0 chmod g+s
# Make the targets of symlinks immutable
find . -type l -print 0 | xargs -0 readlink -z | xargs -0 chattr +i
Way faster.
But there are lots of caveats. Make sure your programs support it. Maybe read the xargs man page.
Personally I skip the middleman when I can with "find ... -exec cmd {} +"
find . -type d -exec chmod g+s {} +
Or even minimise arguments by including a test if the chmod is even needed:
find . -type d \! -perm -g=s -exec chmod g+s {} +
I actually have a script that fixes up permissions, and I was delighted to fit it in a single find invocation which only performs a single stat() on each file in the traversal, and only executes chown/chmod at all for files that need change:
There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1].
If you have to output in a shellscript in this format, use printf %q
from man printf:
%q ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable
characters with the proposed POSIX $'' syntax.
It is just $'<nasty ansi-c escaped chars>'
$ touch $'\nHello\tWorld\n'
$ ls
One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset.
One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though.
Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in.
If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering.
Thanks for sharing, I love stuff like this. I get frustrated at mtimes being munged. MacOS finder is terrible about this.
If anyone is interested, I wrote a more generalized tool for storing metadata called storetouch[1] which takes a snapshot of file's mtimes, and stores them in a nicely formatted file for convenient restoration. You can run it's output by itself with bash, or easily parse it's output.
I found that the commit or author date's did not always make sense to represent the true age of a file. Especially for non-source-code documents. For example, a repo of scanned documents. Or a git-annex repo of podcasts going back 20 years.
Instead, I treat the commit date as the commit date, and I have a versioned sidecar that stores the actual original mtimes of the files.
It can also handle all valid POSIX filenames (a surprising amount of utilities don't, including tools like git-restore-mtime, last time I checked).
If you're already using a tool like git-annex or datalad though, they have the ability to store metadata for each file as well.
I would also suggest checking out git-store-meta[2] which was the original inspiration. It stores more than just mtime, but it is more closely tied to git.
The mtimes aren't being munged. mtimes are filesystem metadata that reflect when the file was changed; trying to use them as version control information (or application level metadata) is wrong. For example, mtimes are used by programs that cache file data, like file indexing programs; backdating mtimes will break everything that uses mtimes for their intended purpose.
Yep, and a lot of backup software will use them (along with file size, and other stat() fields) to determine if a file has changed, so backdating them can actually lead to silently stale backups
Sometimes search indexing and build tools get it wrong, so if you know what you're doing, then tools like this allow you to provide an override.
Just browsing an old photo collection without making any changes can cause a 15 year old file to show up as a "new file", even though it's contents have not changed. This is especially true with media files that the os might take it upon itself to regenerate metadata for.
Or you may wish to store an accurate snapshot of the timestamps of cached or intermediate build artifacts.
Both Finder and explorer.exe will mung mtimes from casual browsing, usually due to metadata updates.
It is a common enough problem that there are utilities to bind mtimes to files.
IMO mtime the de-facto file timetsamp metadata, since it is most widely supported. The other file timestamp metadata I found were not as useful or portable.
And people use git for storing more than just source code.
I personally find it very useful to be able to clone my podcast index and have accurate timestamps not just from the commit, but from the file itself.
Timestamps can be important for a collections of files, but they are fragile and often an afterthought.
You can already use it with git-annex to store binaries using the git-annex-remote-git-aafs[1] special remote.
Although I would be careful and make sure you understand what it is doing to your branch namespace. Even though in the worst case it would not save any space over directly committing binaries, they are in orphan branches that can be pruned without rewriting history.
But even so, you can just use any number of git-annex special remotes to bypass using git for sharing files.
They may eventually add first-party support for git-annex. But nothing is stopping you from using it now.
I will probably give bcachefs on one of my machines.
I started using btrfs back when it was way less stable, but that instability was more than offset by the other features.
The last time I lost a btrfs filesystem was four years ago when it was less stable, but it did not matter because it only amounted to about an hour of logs lost, and I reconstructed it in minutes. So btrfs is a must on any embedded devices. It frees me to use cheaper media, because I can have two copies of all data and metadata, and compression, and fine grained external snapshots.
Zfs is great for making big complex arrays and if you plan out how your are storing your data ahead of time. But it has a steeper learning curve and is more seperate from the underlying OS.
Btrfs fills this perfect niche for me of being a drop in replacement for ext4, and being really flexible and offering some of the nicer features of zfs. (differential snapshots, checksumming, compression)
So I'm excited to try bachefs as an alternative to btrfs.
This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots.
I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.
I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.
I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.
My setup is that I only have subvolumes for root, and (for now) my Doom Emacs config. Whenever I do a system upgrade, or a Doom Emacs upgrade, I manually take a snapshot of the appropriate subvolume first, just in case anything goes wrong. And then I only keep a max of 7 or so snapshots before I go through and delete the old ones.
Has worked well for me so far and saved my butt a few times, and since I don't have a bunch of subvolumes to manage, I'm not overwhelmed. It's nice and simple, and I need nothing other than the default `btrfs` CLI utility.
That's what's great about btrfs. I started using it that way too. It was nice to be able to take instant snapshots of my fs. I just amended my existing backup scripts to act on temporary snapshots instead. Then, I started dividing things up into subvolumes.
I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.
I also highly suggest reflinks. Once you start using them you will use them all the time.
I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.
The computer which run iVentoy must be in the same LAN with them.
Besides, there must no other DHCP servers in the same LAN.
It can coexist with an existing DHCP server. That means you can just start it up on your own network without any modifications, and start netbooting machines.
It turns out that multiple DHCP servers can co-exist on the same network segment. So I just have a second dhcp server solely for a pxe bootstrapper. At the time, at least the initial download must be tftp, but I used a stripped down ipxe stub that took advantage of the universal undi network drivers, and then loaded the rest (usually syslinux) over a fast HTTPS connection.
From there, iSCSI was no problem, and I could also use tools like plop, memdisk, or grub4dos, as well as Windows 7/8/10 with wimboot.
I remember for some distros like Knoppix, we would use an NBI driver to just mount the iso directly over the network, which could be far simpler and more performant than NFS.
EDIT: To be honest, it is simple enough that it doesn't have to use Docker. It is mostly dnsmasq and PHP.
We're like frogs that have been stoking the fuel of our own pot.
It used to be that DRM was considered to be in conflict with the browser, because it was not acting on behalf of the user. If you must have DRM, then it is on the platform to shoehorn it in through an external plugin, like Silverlight.
When Firefox adopted EME extensions, I knew it was the beginning of the end; they were rolling out the red carpet for DRM. If we make DRM a switch that can simply be thrown, then it will become the norm, not the exception. And there have been proposals for years to DRM fonts and other absurdities. If a company insists on using DRM, then they should have to shoulder the burden of doing something that a browser was never support to support.
The nightmare that we're racing toward is you will only be permitted to cache a trickle of video at a time and your TPM attestation hardware must include a token in every HTTP request. Your browser will just be a software cablebox.
They aren't happy about URLs either and would love to require that if you want to share a reference to something, you have to do it on their terms, like generating a url in their app with a hash that expires and limited in how many times it can be viewed. I'm sure influencers will still have the privilege of unlimited sharing.
They've been slowly rolling this infrastructure out for the last decade. These are not isolated inconveniences, these are coffin nails.