No can’t run custom OS as far as I know; without some seriously hackery. Admittedly UniFi NAS software currently is not nearly as feature rich and enterprise as TrueNAS for example, but it’s all very new. Ubiquiti has had a track record lately of iterating very quickly adding more and more enterprise features into their products. I expect to see more hardware offerings in the UNAS line soon as well. Highly recommend Ubiquiti! I have a complete setup at my house with their switches, routers, APs, and protect cameras. It’s awesome!
I assume the UniFi NAS isn't meant for running Plex, for example. Most people seem to want a general purpose mixed media server, which can handle a bunch of data but where a good chunk of that data is movies/music/books that they want to stream. A good percentage of people probably also want to run docker containers. The $500 for (was it 7?) bays of storage is great, but probably mainly functions as a true NAS, so maybe not what a lot of people want. (I think the term "NAS" is a bit overused and can be deceptive for newer users entering the market)