Creating the environments observed by the gaze in a photo in 3D: an operation that is still imperfect but possible
We are not yet at the levels of Minority Report or Vanilla Sky but the die is cast. Some researchers have in fact managed to achieve the 3D reconstruction of the environments of a room starting from the reflection of walls and objects in the eyes of a person portrayed in a photo. Science fiction? The incipit is that but the reality is much closer than you think. This is all thanks to Neural Radiance Field (NeRF) technology, which is starting to show some incredible capabilities in 3D reconstruction of 2D images and video.
The work is due to researchers at the University of Maryland, who have demonstrated the possibility of acquiring various high-definition images of a person moving in a room, to enlarge the reflections in the corneas, flip them, remove the color and details that irises could have added and ultimately process the raw information for the 3D reconstruction of the surrounding environments.
As anticipated, these are not very high resolution 3D models (also because they are, to all intents and purposes, a 3D reconstruction): we can distinguish one object from another but not in detail and, above all, not without resorting to a some photo editing. The point is: is there a need for such technology? Maybe not today but for a crime scene in 2030 maybe yeswho can tell.
The researchers created a real-world scenario to test a few solutions to achieve their goal. The first point: zoom in on the reflections of the eyes from the video clips of Miley Cyrus and Lady Gaga, hoping to take advantage of the quality close-up vision and favorable lighting conditions. Unfortunately, however, the resolution was not high enough, and the maximum result they have come up with is that Miley Cyrus was looking at a series of lights… Nothing extraordinary or unreliable. Then the turning point: treating those images in three dimensions and not just two.
3D reconstruction of the image, like a rendering made starting from the 2D vision
Within the image are all kinds of eye artifacts, the complexity of the textures of the iris and identifiable low-resolution reflections captured at every instant. To remove the iris from the images, the researchers performed texture decomposition by training AI software that learns how the unique and unrepeatable texture of an iris is composed to eliminate it.
Using the geometry of the cornea, which is approximately the same in all adults, calculations were made to track exactly where a person’s eyes in a photo are looking. This also allows you to determine the camera angle by plotting image coordinates on the curved geometry and setting a viewing direction for the NeRF technology to be used later to proceed with the 3D reconstruction.
Despite subtle inaccuracies in corneal position and geometric estimates, the method was effective in scene reconstruction. Area lights placed to either side of the person (out of frame) were used to illuminate the object of interest while the person being photographed was asked to move within the camera’s field of view as more images were captured.
More than the application itself, the main consequence of this experiment concerns the steps taken by Neural Radiance Field (NeRF) technology, which is advancing with astounding results. Suffice it to say that there are services for the end user, such as LumaLabsAI, which use this technique to give different effects to a simple video taken with a mobile phone.
Among other applications, UC Berkeley has combined a NeRF modeling network with a language model to create the “Language Embedded Radiance Field” or LERF model, which allows you to search for particular elements in a 3D scene using the natural language.
Imagining a future in which individual data are no longer self-referential information but pieces of an open ecosystem, which can always be recalled and analyzed for an infinite number of reasons, is as realistic as it is worrying.
breaking latest news © (Article protected by copyright)