Scientists Develop AI Capable of Recreating Images From Brain Scans

Researchers have developed a novel artificial intelligence system that can recreate images a person is observing by analyzing their brain scans. The AI, named Brain-IT, achieved remarkable accuracy in reproducing pictures that it had not previously encountered, marking a significant advancement in the field of neural decoding.

During a study, participants were shown a variety of images, including scenes like a baseball game, a dog in a car, and a snowy landscape. Brain-IT processed the brain activity patterns corresponding to these images and generated reconstructions. Professor Michal Irani of the Weizmann Institute of Science stated that while existing models can translate brain activity into images with reasonable semantic accuracy, they often falter on basic visual elements like composition and color. The newly developed model, however, excels in reconstructing both the content and detailed features of the original images.

IMAGE4 A key breakthrough of Brain-IT is its significantly reduced training time. Previous AI models required approximately 40 hours of brain scan data to learn to interpret a new individual's neural activity. In contrast, Brain-IT requires only about one hour of data to achieve comparable results. To demonstrate this efficiency, the team compared reconstructions generated by Brain-IT trained for one hour against those trained for 40 hours, finding the outcomes to be remarkably similar.IMAGE6

The system was developed by feeding it thousands of brain scans from eight volunteers who were looking at different images. This training process enabled the AI to learn the correlations between specific neural activity patterns and visual elements such as colors, shapes, and objects. The AI became so adept that it could predict the appearance of a brain scan based on an image.

By consolidating data from multiple studies, the researchers identified specific brain regions that consistently perform similar image-processing functions across different individuals. For instance, certain areas responded to images of food, while others were activated by pictures related to sports. Professor Irani noted that the AI's encoder naturally identified 128 functional regions shared by all participants, some of which are known to neuroscientists, while others are newly discovered. The research even uncovered a division of labor within the brain region responsible for processing images of places, with distinct parts responding to indoor versus outdoor scenes.

IMAGE2 When presented with new brain scans, the AI was able to generate highly accurate representations of the original viewed images. The team's comparisons with other AI programs also revealed that Brain-IT produced more precise reconstructions. IMAGE3

Professor Irani's lab is exploring the extension of these methods to decode auditory information and is investigating the potential for decoding video, which presents greater challenges due to the rapid succession of images. The ultimate goal, though still distant, is the possibility of reading dreams, should obstacles in decoding rapid visual changes during sleep be overcome.

IMAGE1 Furthermore, the team is working on adapting these techniques for brain activity recorded using electroencephalography (EEG), a non-invasive method that measures electrical signals from the scalp. As AI models advance, scientists anticipate that interpreting EEG data will become increasingly feasible, offering an alternative to more complex methods like fMRI scans.

IMAGE5 The findings were presented at the Cognitive Computational Neuroscience conference in New York last month.