Very interesting. I've implemented something similar. It evolved out of the co-browsing solution I developed for the company I work for.
The solution uses mqtt. Clients subscribe to a topic on the server, and the server publishes patches to update the view. Patches can be incremental (patch against the last frame), cumulative (patch agains the last keyframe) or a new keyframe. It allows for server side rendered views. Multiple clients can subscribe to the same view and keep in sync. See: https://github.com/mmzeeman/zotonic_mod_teleview
The best way to do pub/sub on the web with a standard protocol is MQTT (https://mqtt.org). It supports websockets, it scales, supports authentication, can handle unreliable networks.
We use it exclusively for the soon to be released 1.0 version of Zotonic. See: https://test.zotonic.com (running on a 4.99 euro Hetzner vps).
We developed an independent support javascript library called Cotonic (https://cotonic.org) to handle pub/sub via MQTT. This library can also connect to other compliant MQTT brokers. Because MQTT is fairly simple protocol, it is fairly easy to integrate in existing frameworks. Here is an example chat application (https://cotonic.org/examples/chat) which uses the open Eclipse broker.
> The best way to do pub/sub on the web with a standard protocol is MQTT
Strong disagree. MQTT has no place in a world with WebSockets. MQTT knowledge is so esoteric in comparison. Maybe it's the Haskell of transport protocols: really friggin smart, but not if you're trying to be useful to society at large.
MQTT is a proven protocol. It's design started started more than 20 years ago. The protocol was designed to be easy to implement. It is simple to implement if you want. There are multiple brokers, and client libraries available. So why invent a new protocol, when an open, standardised protocol already exists.
Open pages in a browser are not that different from IoT devices.
> So why invent a new protocol, when an open, standardised protocol already exists.
Because despite being a dinosaur, it still only has middling adoption. It exists in a particular niche of a particular niche of computing.
HTTP, in contrast, is English to MQTT's German. One is just infinitely more common and accessible to the majority of the world. So, if you're serious, use it.
MQTT is neither "esoteric" or "not useful to society at large" nor is WebSockets vs MQTT even a useful contrast, since WebSockets to a large degree addresses a different level of the stack (and indeed "MQTT over WebSockets" is a common way of deploying it).
Did research on SSE a short while ago. Found out that the mimetype "text/event-stream" was blocked by a couple of anti-virus products. So that was a no-go for us.
It's not blocked. It's just that some very badly written proxies can try to buffer the "whole" response, and SSE is technically a never-ending file.
It's possible to detect that, and fall back to long polling. Send an event immediately after opening a new connection, and see if it arrives at the client within a short timeout. If it doesn't, make your server close the connection after every message sent (connection close will make AV let the response through). The client will reconnect automatically.
Or run:
while(true) alert("antivirus software is worse than malware")
These days I feel like the only way to win against poorly designed antiviruses and firewalls is to—ironically enough—behave like malware and obfuscate what's going on.
Back in the day there was also a huge security hole which could be exploited with the xwd command. If you knew the dns name of a terminal you could get a screendump from it. It was not secured at all.
I worked on a Java program that was a point of sales terminal. All the registers were reasonably beefy linux machines that ran a Java Web Start application and it worked well... except when it didn't.
Sometimes it had threading deadlocks and that was a pain to debug. In particular, we didn't know where it came from. So we installed a handler on ctl-alt-delete that would find the Java process running and do a thread dump from that (so we could find the methods that were locked) and also did a xwd to capture the screen to see what was happening. Sometimes we were able to pick out "this modal alert window spawned 100x copies of itself" by recognizing an unnaturally thick border around it. Just seeing what the state that the register was in was useful too.
The xwd and thread dumps where then stored in a folder that was sent back as part of nightly reports back to the central office where we'd look at them the next morning.
On one hand it was a bit clunky, on the other hand it really beat trying to get a bug report from cashiers working the registers in a timely manner - they just had to remember to do a ctl-alt-delete rather than power cycling the machine and we'd get it reported back to us automatically.
You don’t have to design a new protocol. There is an open protocol which works nice for this use case. It can handle dropping connections, has a nice routing mechanism, it scales pretty well, has different QoS levels, pub/sub, request/response. That protocol is... MQTT v5!
Personally I’m surprised it is not used more often by web devs.
MQTT v5 is fine enough until you have to care about reliability at scale. What I have seen happen is that it turns into a slightly better TCP where you have to build your own primitives.
For example, when you SUB, you get a SUBACK if enabled. However, there is nothing like a SUBFIN (subscribe over/finished/closed) to indicate that the subscription is over or failed. In many ways, pub/sub overcommits in many ways. Now, this all depends on what does SUBACK mean? is it durable? tied to a connection?
An interesting foil is something like rsocket. RSocket has a lot of nice things, but it requires an implementation to be damn near perfect. Paradoxically, rsocket lacks the notion of a SUBACK but it has something like a SUBFIN.
A key challenge with a stream is knowing if it is working. With request-response, you can always time out. With a stream, how does one tell the difference between we broken stream and an inactive stream?
We use this at my place of employment and it’s generally OK except for dealing with reconnects. Also pub/sub request/response was not nearly as smooth as we had hoped and eventually we reverted back to HTTP
