2026-10-09T11:39:30Z https://api.figshare.com/v2/oai

oai:figshare.com:article/34039200 2026-10-01T05:30:48Z category_15 portal_316 item_type_3 month_year_10_2026

Data Sheet 1_MEMOIR-VLM—a multimodal vision-language model for Alzheimer's disease classification and question answering.pdf Sohail Haresh Gidwani (25152636) Tamoghna Chattopadhyay (18948082) Sophia I. Thomopoulos (6376262) Paul M. Thompson (135650) Neuroscience Alzheimer's disease deep learning missing data vision-language model retrieval-augmented generation visual question answering diffusion tensor imaging large language model Introduction<p>Alzheimer's disease (AD) affects an estimated 55 million people worldwide and is projected to nearly double by 2050, creating a need for scalable, non-invasive AI systems that can classify disease stage, support multimodal clinical reasoning, and interact in natural language. Most deep learning work on AD focuses on single-modality classification without natural-language interaction and typically assumes complete input data.</p>Methods<p>We present MEMOIR-VLM (Multimodal Encoder with Missing-modality and Open-ended Inference for Retrieval), a modular two-stage vision-language framework trained and evaluated on 2,363 ADNI subjects. The first stage is a missing-modality-aware encoder that fuses T1-weighted MRI and DTI fractional anisotropy maps with structured clinical scores via cross-attention fusion with stochastic modality dropout, allowing a single set of weights to operate on any of the seven non-empty modality subsets without imputation. After CLIP/InfoNCE contrastive pretraining, the encoder is fine-tuned for multi-task prediction of 3-way diagnosis, binary diagnosis, CDR-SB, age, and sex. The second stage extends the frozen encoder into a retrieval-augmented generation pipeline for case-based reasoning and open-ended visual question answering.</p>Results<p>On held-out ADNI test data, with CDR-SB excluded from the inputs to remove a cognitive-score leak, the encoder achieved 91.3% balanced accuracy for CN vs. dementia and 68.2% balanced accuracy for 3-class diagnosis (macro-F1 = 0.681); age was predicted with a 5.96-year MAE. When the diagnosis field was withheld from retrieved captions, no evaluated LLM exceeded a retrieval-only k-NN majority vote (0.673), which itself closely matched the encoder's 3-class accuracy (0.682). The RAG-VQA layer showed strong text fidelity (BERTScore = 0.894; SBERT cosine similarity = 0.811). Zero-shot external validation on OASIS-3 (1,048 subjects, no diffusion data) retained 78.7% balanced accuracy for CN vs. impaired (AUC = 0.889) using identical ADNI-trained weights with the DTI branch masked.</p>Discussion<p>MEMOIR-VLM combines missing-modality-aware multimodal encoding, multi-task clinical prediction, and retrieval-grounded natural-language reasoning within a single modular framework. The results support using the encoder, optionally with k-NN voting, as the diagnostic component and the RAG-VQA layer as an interpretable natural-language interface that surfaces comparable cases and readable clinical summaries. The external validation results further demonstrate the practical value of the missing-modality mechanism when an imaging modality is unavailable.</p> 2026-10-01T05:30:48Z Dataset Dataset 10.3389/fncom.2026.1902258.s001 https://figshare.com/articles/dataset/Data_Sheet_1_MEMOIR-VLM_a_multimodal_vision-language_model_for_Alzheimer_s_disease_classification_and_question_answering_pdf/34039200 CC BY 4.0