IMGEF: Integrated Multi-Modal Graph-Enhanced Framework
Overview:
IMGEF is an advanced framework designed for generating high-quality radiology reports by integrating multi-modal features through graph-based techniques. This repository provides the implementation for the IMGEF model, which incorporates spatial-aware graph embeddings, multi-modal attention mechanisms, and fusion techniques to ensure state-of-the-art performance on benchmark datasets.
Features
Graph-Based Representations: Incorporates clinical, visual, and textual features using graph-based embeddings to capture spatial and semantic relationships.
Multi-Modal Attention Fusion: Efficient fusion of multi-modal data using advanced attention-based mechanisms.
Optimized for Radiology Reports: Specifically tailored for radiology report generation tasks, addressing long-term feature dependencies and data imbalance.
Requirements
Ensure the following dependencies are installed before running the code:
torch==1.11.0+cu111
python==3.7
torchvision==0.8.2
opencv-python==4.4.0.42
Install dependencies using pip:
Datasets
IMGEF is evaluated on the publicly available IU X-Ray dataset.
-
Download the IU X-Ray dataset from the here.
-
Place the dataset in the following directory structure:
data/
└── iu_xray/
├── images/
├── reports/
└── annotations/
Usage
1. Training the Model
To train the IMGEF model, execute the following command:
python maintrain.py
2. Testing the Model
To test the trained model on the IU X-ray dataset:
checkpoints/imgef_model.pth
3. Evaluation
To evaluate the performance using standard NLG metrics.
Results
IMGEF achieves state-of-the-art performance on the IU X-ray dataset. Detailed results and comparisons are available in our paper.
Acknowledgments
This work is supported by a grant from the Natural Science Foundation of China (Grant No. 62072070).
We would also like to express our gratitude to all the source code contributors, especially the authors of R2GenCMN, whose work inspired parts of this implementation.
Citation
If you find this work helpful, please cite our paper:
For any questions or issues, please feel free to contact us:
Email: