Every few months I see new mind blowing NeRF results, but I wonder if there is a usable project that I could try it myself. Be it open source project, or commercial product.
Also I'm not quite sure what time scales are we talking about, with turning bunch of images to nerf model, and then using that model to generate novel views.. hours on powerful gpu? minutes on cpu?
I went on a trip to Italy last year and took a video while walking around Michelangelo’s David. Even with that relatively poor image quality this let me turn it into a pretty high quality nerf.
They didn't turn HDR mode on, that's the novel aspect of the research. They took SDR photos with different positions and exposure levels, and then merged them into the NeRF.
Before modern sensors with huge dynamic range for a single photo, the "old school" way of getting HDR was to put your camera on a tripod, and take 5+ shots with different exposure levels. This can then be merged in software, but obviously this only works on static scenes and a static camera position.
The HDR-NeRF model lets you move the camera around, you don't need a tripod.
PS: Something I find hilarious is that the paper talks about HDR and shows "HDR-SDR" comparisons... but both pictures are SDR JPG images. It's pathetic how the PC world is pathologically unable to uplift imaging standards while the TV and Camera industries with far smaller R&D budgets are casually sailing past us into the future.
I thought this was essentially how "computational photography" worked, like in the Google Pixel photo app: The camera takes several pictures with different exposure, maybe even from different lenses, and an AI model merges them.
Your PS reminds me of the old ads for those Matrox 30bpp graphics cards, which ‘showed how good these cards looked’ compared with standard graphics cards. Which we watched, obviously, on a computer with a standard graphics card.
Also the TV salesman telling me that a 100Hz panel would look jerky, and showing me a video on a 200Hz screen that allegedly illustrated the difference.
But what advantage does not turning HDR mode on give vs doing so is the question. I guess you could be trying to generate a NeRF with a really old smart phone?
Right now the initial hot industrial application for NeRF's are recovering radiance fields from ordinary video footage. You can have someone just walk around with an ordinary camera or phone, just making sure they have decent coverage. Then back in the studio after a bunch of crunching you get a 3d virtual set you can position a camera in arbitrarily. The quality of this is considerably better than previous approaches that were based on recovering 3d meshes.
The work in the original post can now extend that process to capturing HDR data, which should enable better quality when compositing with other footage or rendered elements. And you don't need some special HDR video camera, just an ordinary one can sample the same thing.
AFAIK other NeRF methods don't let you use HDR data, just LDR images, so you would have to pre-process your HDR photos (tone-mapping) and you wouldn't be able to control exposure afterwards as previous necessary information for doing so would not be present when inferring novel views.
HDR-NeRF allows you to control both exposure and position when synthetizing new images. Your intuition is on the point though, with enough data/compute you could build a method that uses the photos with full dynamic range information from a HDR camera instead of "stitching" LDR photos of varying exposures like this method does.
Use today could be in turning older non-HDR images into HDR NeRFs, but also as a foundation that somebody else will build something completely new on top of. Who knows what that may be.
It lets you synthesize views that you didn’t take with the camera.
I am into making stereograms and this sort of technology would help, for instance maybe the separation of my two cameras is more or less than I wanted so I could make up the views I really wanted.
I see these videos on YT, "walking through Tokyo 4K" stuff like that. It would be cool to have something that can watch the video and generate a street level map with depth of what it saw. You can then go through with a camera head level and navigate it. Can be done but yeah.
The technology would also be killer in real estate markets. Imagine being able to take a video and give potential tenants the ability to see the apartment/house/etc in detail.
I cynically think it wouldn't be a killer, because real estate agents prefer to have photos carefully made with fish-eye lens that makes rooms look bigger.
How do they decide where to use the little letters in these acronyms? Seems like it should either be HDR-NRF or something like HiDR-NeRF. Come on folks. Naming consistency, please.
NeRF was already standardized in the literature by people who are not these authors [0] and HDR is also already standardized by people who are not these authors [1]. Go with the flow unless you have a very good reason not to.
HDR is probably HDR and not some other clever name because there was no obvious clever name with an easy pronunciation. Whereas NRF is just so close to NeRF -- an actual word people are familiar with -- it's begging to be pronounced that way.
I will complain because "HDR" is often usually used to refer to the process of /removing/ HDRness from data, ie, tone-mapping, so you can display it on an SDR display. Or so you can make one of those /r/shittyHDR images that looks like it's been deep fried.
Tone mapping was all the rage 10-20 years ago. There was even a name for it (the X look) but I forgot which unfortunate photographer's name was associated with it.
> Tone mapping is a technique used in image processing and computer graphics to map one set of colors to another to approximate the appearance of high-dynamic-range images in a medium that has a more limited dynamic range. Print-outs, CRT or LCD monitors, and projectors all have a limited dynamic range that is inadequate to reproduce the full range of light intensities present in natural scenes. Tone mapping addresses the problem of strong contrast reduction from the scene radiance to the displayable range while preserving the image details and color appearance important to appreciate the original scene content.
> […]
> The goals of tone mapping can be differently stated depending on the particular application. In some cases producing just aesthetically pleasing images is the main goal, while other applications might emphasize reproducing as many image details as possible, or maximizing the image contrast. The goal in realistic rendering applications might be to obtain a perceptual match between a real scene and a displayed image even though the display device is not able to reproduce the full range of luminance values.
Yes, the vowel matches, but you could imagine the original authors had chosen NRF, pronounced "en ar eff". But the fact that the e from Neural was there, and the word nerf was there, just a perfect alignment.
I personally appreciate abbreviations that easily roll of the tongue and seem like words on their own. I believe it's important to be able to talk fluently about these things, to better relate to and understand them.
Also I'm not quite sure what time scales are we talking about, with turning bunch of images to nerf model, and then using that model to generate novel views.. hours on powerful gpu? minutes on cpu?