Great interview. The thought that stands out for me is the repeated assertion that fantasy UIs must serve the narrative first. That's analogous to the way a lot of people design UIs for real products in that they tell "user stories."
However, there will always be tension between that narrative-first approach and the real world where narratives don't exist. A real user, trying to use a real interface, just needs it to work. There's no audience, no dramatic tension, no character arc, etc. It just serves their needs in the moment or it doesn't.
In a story, the fantasy UI really is a pixel-deep metaphor substituting for the technological capabilities available to the characters. "Access denied" isn't really an interface, it's a visual way to communicate a complex plot point, just like all of the other crafted visuals in the movie. Nobody ever has to puzzle over a bad interface for the same reason nobody ever has to go to the bathroom.
But real UIs are more than one pixel deep. They're the Rosetta stone that translates between human thought and machine logic.
So when Minority Report makes up a gesture-based interface it's because they have to show the audience something plausible that they've never seen before. However, when a designer makes a real gesture-based interface it's because there's something their users can't do with the interface they're used to.
My favorite movie of all time is called Stranger Than Paradise by a director named Jim Jarmusch, in part because of its original treatment of these issues. The film is peppered with subtle little moments where, for example, a kitchen cabinet doesn't close correctly, or someone doesn't hear the person they're talking to correctly and asks them to repeat themselves.
Neat. Kind of like how there's a whole website tvtropes.com devoted to collecting examples of all the different shortcuts used to tell stories. After a while, the trends in the material become interesting in their own right. Someone who's seen a lot of movies will see that something went wrong and wonder if it's a joke intentionally poking fun at the movie structure itself. Someone who hasn't seen a lot of movies probably won't even notice. I read somewhere that when an isolated tribe in Africa was shown pictures and asked to comment on them it took the researchers a while to figure out that the people weren't interpreting the pictures as representations of other things, but as flat, abstract blobs of colors. They literally didn't have enough experience with photographs to use them correctly. Whereas someone with tons of experience can intentionally misuse photographs to communicate one a different level.
Bo Burnham's standup special has some meta jokes like this. I won't spoil them in case you haven't seen it yet.
> However, when a designer makes a real gesture-based interface it's because there's something their users can't do with the interface they're used to.
You're right when the designer is good. A bad designer might make a real gesture-based interface just because he saw Minority Report and thought that it was cool.
There aren't many good designers working today (or maybe their work is being hindered in companies). If you look at contemporary UIs, they're shallow. Optimized for looking nice and selling things, instead of being ergonomic and giving users efficient ways to achieve their goals.
Haha, yeah, I actually rewrote that sentence a couple times because I was trying to phrase it in a way that wasn't wrong, but also got me out of having to address tangential topics like exploration, creativity, and art.
I think movies with "real" futuristic interfaces would be boring. For example the Minority Report movie with wifi would have none of the disks being inserted. Playback would happen in people's heads. Nothing would really be moving. Also, Tom Cruise wouldn't be able to just go back to his base and still get in. Also, they wouldn't make such silly mistakes letting "the hero" have so much leeway. He'd just be quickly captured and that's it.
Yeah, the first thing to go in a good story is good security. Security is designed to make sure nothing interesting ever happens. At best it slows everything down and at worst it just outright forbids everything. That's poison to a narrative.
Yeah, I think having a display that isn't obviously part of your outfit will be a really big enabler of wearable technologies. That and flexible printed circuits that can survive dry cleaning or a washing machine.
One of the reasons Glass failed was because it was about as in-your-face as possible. Resistive screens also failed in the same sense because having to pull a stylus out to use your phone is aggressively technological.
In contrast, touch is a really casual interface, and though the touch UIs that I've used all had warts it was at least possible for anyone to grasp the basic mechanics of the interface. In my mind a wearable device that is obviously a wearable device isn't going to be as successful as clothing that monitors your heart rate or SpO2 because you have to consciously interact with the device. If you fill your gym bag with smart clothes then you just wear them like you would wear gym clothes.
Yeah. The way I think of this is, let's start from (my subjective) aesthetic first principles. Take an abstract human body, and look at its poses and movements: standing, sitting (and other forms of resting), walking, running.
A smartphone is a complete failure from that perspective: looking down and pecking at a screen? No way. Now, holding and looking at something is a natural enough movement, but chronically craning your neck down with such focus and duration? I doubt it. A smartwatch is a bit more natural, but to bring up subjective aesthetics again, I'm not too comfortable with the concept of adults wearing digital watches.
Google Glass was a step in the right direction in my opinion--you can interact with a UI while looking straight ahead--but as you said, the camera and other aspects weirded people out (there may be a future in contact lenses with similar tech?) Voice interfaces like Alexa are definitely in line with my view.
I don't think it's about the pose. It's about attention. To interact with a system we have to focus on it and that means we have to take our focus off of everything else. Computers are notoriously bad at understanding us if we aren't focused on them exclusively.
That's why so many people are trying to get voice interaction right. It's the most natural way to split our attention.
A lot of this IoT nonsense is trying to discover a few corner cases where computers can do sensible things without a human needing to specifically focus on them. Like automatically reordering cereal, or unlocking your phone when you get close, or adjusting the temperature when you leave the house.
> Resistive screens also failed in the same sense because having to pull a stylus out to use your phone is aggressively technological.
Resistive screens worked fine with fingertips, too. In fact they worked perfectly well with gloves, pens, etc., unlike modern capacitive screens. They failed because they were less sensitive and less accurate.
I enjoyed looking at some of the interfaces developed to interact with scripts in SecondLife - often they had animations that hooked into the avatar to perform, to give feedback. However most appeared to draw their inspiration from cyberpunk, rather than science fiction or showing new interfaces.
However, there will always be tension between that narrative-first approach and the real world where narratives don't exist. A real user, trying to use a real interface, just needs it to work. There's no audience, no dramatic tension, no character arc, etc. It just serves their needs in the moment or it doesn't.
In a story, the fantasy UI really is a pixel-deep metaphor substituting for the technological capabilities available to the characters. "Access denied" isn't really an interface, it's a visual way to communicate a complex plot point, just like all of the other crafted visuals in the movie. Nobody ever has to puzzle over a bad interface for the same reason nobody ever has to go to the bathroom.
But real UIs are more than one pixel deep. They're the Rosetta stone that translates between human thought and machine logic.
So when Minority Report makes up a gesture-based interface it's because they have to show the audience something plausible that they've never seen before. However, when a designer makes a real gesture-based interface it's because there's something their users can't do with the interface they're used to.