Well it isn't. First of all it's the obvious software deficiencies - it'll still boot a heavy OS (boot times can exceed a few minutes from col start), even operations like switching HDMI inputs take significantly longer.
Also several manufacturers (I know of some Samsung versions and Phillips) will position web services in first-level UX navigation making those TVs annoying to use as dumb TVs. There are even some models that will constantly nag you to connect them to the internet!
We managed to get zap time < 300ms on our IPTV platform, but that required A LOT of additional work configuring encoders, networking equipment and messing around with STB firmwares.
There's bunch of places between the video source and you that add their own buffers (not to mention the actual I-frame issues). Usually the most critical are encoders themselves (need to generate closed GOPs and a lot of commercial expensive hardware stuff doesn't give you enough control) and the player software on STBs (those usually have large buffers pre-set).
In the end it's rather large amount of work most IPTV (and digital cable) providers don't care about because... hey... where are you gonna go if you're unhappy with their service? :)
It's probably because of the span between consecutive I-frames (or intra-block sweeps if not using I-frames) in the video stream, but audio should switch "instantaneously" (within 1 frame).
Hmm, if that were the case, wouldn't the time spent waiting be variable? That would mean that you'd sometimes get instant switches, when the I-frame was the next frame you received after the switch.
I have an LG OLED TV. Never connected it to the Internet. Instant-on with the Apple TV, haven't interacted with the Smart TV OS at all and doesn't seem to slow it down.
It can be hard to unplug it. I have a 2014 Samsung Smart TV. I didn't want the smart stuff but there wasn't much option. When I first got it I gave it my wifi password just to check it out, confirmed that I'd rather just use my HTPC and turned off wifi.
Lo and behold every few months something would happen which would make it clear that it was connected to the internet. I'd get banner ads. I'd get notices from the mothership about important updates. That sort of thing. Each time I'd discover that the wifi was back on and each time they'd have hidden how to remove it somewhere else.
Eventually I just changed my wifi password and that seemed to have done the trick.
I have a Sony Bravia Smart TV. The only thing I use on it is the screen, the power button, and the volume control.
Everything else is handled by the Internet connected Amazon Fire TV stick I plug into it. It is effectively a dumb TV and will not get infected by malware.
I have an LG with WebOS and try to use my TV the same as you. I don't have it connected to wifi so it's unlikely that it will get infected with malware anytime soon, but it did crash yesterday when I turned it off at the end of the night. I had to pull out the power plug to turn it off.
That's one thing that never happened to me with a (truly) dumb TV and I've only owned this one for a month. I sure hope the CPU hardware running this OS doesn't degrade as quickly as normal consumer PC hardware and I can expect it to run as quick and reliably as it does now in ten years from now (like the TV it replaced did).
I would agree odds are low of something that insidious... but it's hard to say "it'd be impossible for my TV to get hacked" when even light bulbs are getting attacked.
I wonder if you could make a filter of sorts that blocked the ethernet signals while still allowing audio/video? Sort of like those USB "condoms" that block data but allow power.