I wonder if the problem isn't a misaligned paradigm for just what a codec does. Right now, codecs exist to take bytestreams and fill frames at a certain rate. They're a bitstream-to-framerate device.
What if instead of delivering to a frame during a certain time period, the codec had to instantly deliver whatever it could, but only geared to the actual acuity of the users retina. Seems like there would be less information, you would never have lag, and all optimization would be around the detail of data delivered, not framerate or screensize.
I also birthday parties and Bar Mitzvahs, folks. I'm here all week. (I figure it couldn't hurt to throw crazy ideas out there. Every now and then a crazy idea actually amounts to something. Coding is cool because it not only let you solve problems, sometimes it lets you change the universe the problem lives in. Good luck, John!)
But the problem is to display something for a moving angle, not display something for everything. As best I can tell, it is a different problem.
I want to restate what I'm hearing so you can correct me. You are saying something like "Gee, if we could show any retinally-matched conic the user's pointing their eyeball at? To do that we'd have to have the entire scene rendered anyway"
I'm saying you can't do it now. It hurts when I do that. So stop doing that. Instead of trying to render stereo scenes quicker, try to render a moving conic real-time quicker. When the codec runs, it's optimizing areas of a frame changing over time. If instead it optimized possible vision movement paths, you'd end up solving the framerate and resolution problems. Then you could concentrate on optimizing the codec in a different way than people are currently trying to optimize it. It couldn't hurt, and there may be opportunities for consolidation if you look at the problem as visual-movement-path-rendering instead of frame rendering. I don't know. I just know it's a problem now. Set your constraints differently and optimize along a different line. Sometimes that works.
Does that explain the differences here? Or do you want me to start spec'ing out what I mean by retinal-path codec? At some point this will a bit over the top.
ADD: I'll say it a little differently. Our constraint now is "how fast can you render this frame". Codecs are built for rendering frames on screens where people watch movies and play games. But that's not the world we're trying to solve a problem in. They may be configured the wrong way. Instead, require that anything the eyeball is looking at should render consistently in, say 5ms.
This sounds like the same thing, until you realize that the eyeball can't look at everything at the same time. Different parts of the image are temporally separated. So if I update the image in back of your head once every 200ms? You're not going to know. As far you're concerned, it's all instant.
I want to predict the temporal cost of focal path movement, then optimize the stream based on those temporal "funnels" I guess you'd call them. This is in opposition to looking at segments of the screen all equally and optimizing across frame changes. I don't care about frame changes. All I care about is how fast the eyeball can get from one spot to another -- and that's finite. If I can create the image I want instantly wherever the eyeball is looking, framerate and resolution issues are a non-sequiter. (This would also probably scale to more dense displays easier, but I'm just guessing)
I wonder if the problem isn't a misaligned paradigm for just what a codec does. Right now, codecs exist to take bytestreams and fill frames at a certain rate. They're a bitstream-to-framerate device.
What if instead of delivering to a frame during a certain time period, the codec had to instantly deliver whatever it could, but only geared to the actual acuity of the users retina. Seems like there would be less information, you would never have lag, and all optimization would be around the detail of data delivered, not framerate or screensize.
I also birthday parties and Bar Mitzvahs, folks. I'm here all week. (I figure it couldn't hurt to throw crazy ideas out there. Every now and then a crazy idea actually amounts to something. Coding is cool because it not only let you solve problems, sometimes it lets you change the universe the problem lives in. Good luck, John!)