Mind VideoThought to video AI Tool

Mind-Video's two-module pipeline starts with the first module, which concentrates on learning general visual fMRI features via unsupervised learning with masked brain modeling and spatiotemporal attention. This module distills semantic-related features using multimodal contrastive learning with an annotated dataset. Then, the second module fine-tunes these learned features by co-training with an augmented stable diffusion model that is specifically tailored for video generation under fMRI guidance.

In Mind-Video, the semantic-related features are distilled using the multimodality of the annotated dataset. This stage involves training the fMRI encoder in the CLIP space with contrastive learning.

The Stable Diffusion model in Mind-Video plays a crucial role in guiding the video generation. Following the learning of general and semantic-related features from the fMRI data in the first module, the second module fine-tunes these features by co-training with an augmented stable diffusion model. This process specifically focuses on guiding the generation of videos under the influence of fMRI data.

Throughout its training stages, the fMRI encoder in Mind-Video shows progressive improvement in assimilating nuanced semantic information. The encoder learns brain features in multiple stages and shows an increased attention to higher cognitive networks and decreased focus on the visual cortex over time, demonstrating its progressive learning ability.

When compared with state-of-the-art approaches, Mind-Video demonstrated superior results. It achieved an accuracy of 85% in semantic metrics and 0.19 in SSIM, a measure of the structural similarity between the reconstructed video and the original, outperforming the previous best approaches by 45%.

The attention analysis of the transformers decoding fMRI data in Mind-Video showed a dominance of the visual cortex in processing visual spatiotemporal information. However, higher cognitive networks, such as the dorsal attention network and the default mode network, were also found to contribute to the visual perception process.

Mind-Video ensures generation consistency in its process by meticulously preserving the dynamics of the scene within one fMRI frame while enhancing the generation consistency. This equilibrium is critical for accurate and stable reconstruction over one fMRI time frame.

Mind-Video utilizes data from the Human Connectome Project as it provides large-scale fMRI data. This comprehensive set of brain imaging data aids in the effective analysis, learning, and reconstruction of visual experiences from brain recordings.

The contributors to the development of Mind-Video include Zijiao Chen, Jiaxin Qing, and Helen Zhou from the National University of Singapore and the Chinese University of Hong Kong as well as collaborators from the Centre for Sleep and Cognition and the Centre for Translational Magnetic Resonance Research. The tool also acknowledges supporters such as the Human Connectome Project, Prof. Zhongming Liu, Dr. Haiguang Wen, the Stable Diffusion team, and the Tune-a-Video team.

Mind-Video aims to address the challenge of recovering continuous visual experiences in video form from non-invasive brain recordings. This was the primary motivation for its development. The research gap it aims to fill involves overcoming the time lag in the hemodynamic response for processing dynamic neural activities and enhancing the generation consistency while ensuring the dynamics of the scene within one fMRI frame are preserved.

Mind-Video's brain decoding pipeline is made flexible and adaptable through its decoupling into two modules. These are the fMRI encoder and the augmented stable diffusion model, which are trained separately and then fine-tuned together. This design allows the encoder to progressively learn brain features through multiple stages, resulting in a flexible and adaptable pipeline.

Mind-Video achieves high semantic accuracy through a comprehensive learning and fine-tuning process. The encoder learns brain features in multiple stages, building from general visual fMRI features to more semantic-related characteristics. The augmented stable diffusion model then fine-tunes these features, guided by the fMRI data. This results in a recovered video with high semantic accuracy, including motions and scene dynamics.

IDK

The multimodal contrastive learning in Mind-Video serves to distill semantic-related features from the general visual fMRI features learned via unsupervised learning. It utilizes the multimodality of the annotated dataset, training the fMRI encoder in the CLIP space to focus on these essential semantics.

The attention analysis of the transformers decoding fMRI data in Mind-Video reveals that the visual cortex is dominant in processing visual spatiotemporal information. It also shows a hierarchical nature of the encoder's layers in extracting visual features—initial layers focus on structural information, while deeper layers shift towards learning more abstract visual features. Finally, the fMRI encoder demonstrates progressive improvement in assimilating more nuanced, semantic information throughout its training stages.

The code for Mind-Video can be accessed via [this GitHub repository](https://github.com/jqin4749/MindVideo).

Yes, the two-module pipeline that forms the core of Mind-Video—consisting of an fMRI encoder and an augmented stable diffusion model— is designed to be flexible and adaptable for fine-tuning according to specific needs. They are trained separately and can be fine-tuned together, offering a high degree of customization.