“Mind Reading”: Input for Stable Diffusion directly from the brain

With the help of AI, the computational effort required to interpret brain scans can be significantly reduced. The idea is not entirely new: many research groups around the world have already used AI and machine learning to write back the mental content of brain scans. Some researchers have had amazing success, such as Tom Mitchell and Marcel Just from Carnegie Mellon University, who read imaginary words from brain scans in 2008. On closer inspection, however, the method only works within very limited test conditions.

The development of deep learning seemed to give new impetus to research. However, these models first had to be trained on large data sets. And even then, the reconstructed images “similar” to the viewed images at best, as Guohua Shen from the ATR Lab in Tokyo writes in a paper from 2019 – the corresponding image shows blurry schemes at best.

Brain scans as input for stable diffusion

Yu Tagaki from Osaka University and Shinji Nishimoto from the National Institute of Information and Communications Technology Japan now use stable diffusion to reduce the computational effort for such an image reconstruction and to drastically increase the quality of the output. They used images from functional magnetic resonance imaging (fMRI) – the method measures brain activity based on oxygen consumption – to generate inputs for stable diffusion. They then used the power of generative AI to drive the output with image-to-image and a text description.

When people look at a photograph, the temporal lobes primarily register information about the image’s content (people, objects, or landscapes), while the occipital lobe primarily registers information about the layout and perspective, such as the scale and position of the content.

To obtain the text and image input for stable diffusion from the fMRI images, the researchers trained linear models on images from a dataset from the University of Minnesota. The Natural Scenes Dataset (NSD) contains brain scans from eight participants who each viewed 10,000 landscape photos, associated with the images viewed. From the data from the occipital lobes, the researchers generated a compressed image file modified by their AI model. They generated text from the data from the temporal lobes – also using information from the image descriptions during training. They then used image and text as input for the standard model of Stable Diffusion.

Test with picture of teddy bear

To test how well the software works, the researchers used a portion of the data set that they hadn’t used for training, including an image of a teddy bear, an airplane, a clock, and a train. Tagaki and his colleague emphasize that this is “not mind reading”. However, the results are surprisingly good. The contours of the objects on the generated images are easy to recognize and are therefore very similar to the original images (see figure below).

Nishimoto hopes that with further refinement, the technology could be used to intercept dreams or allow scientists to understand how animals perceive reality. However, the AI ​​system was only tested on the brain scans of the four people who also provided the training brain scans. Extending it to other people would require retraining the system on their brain scans.

See also:

(wst)