Merge pull request #395 from Emory-HITI/dev

Merge to main: since Dec 2022

.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"

.gitignore

@@ -10,4 +10,4 @@ htmlcov
 .coverage
 coverage.xml
 **/*.log
-/tests/data
+/tests/data

Niffler.egg-info/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.1
+Name: Niffler
+Version: 0.9.3
+Summary: Niffler: A DICOM Framework for Machine Learning and Processing Pipelines.
+Home-page: https://github.com/Emory-HITI/Niffler
+Author: Soham Mulye
+Author-email: [email protected]
+Project-URL: Bug Tracker, https://github.com/Emory-HITI/Niffler/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+License-File: LICENSE
+
+# Niffler: A DICOM Framework for Machine Learning and Processing Pipelines.
+
+Niffler is a lightweight framework to facilitate executing machine learning pipelines and processing workflows on DICOM images and metadata. Niffler facilitates efficient transfer of DICOM images on-demand and real-time from PACS to the research environments. Niffler is also integrated with the radiology information system (RIS) to get clinical data in real-time. The DICOM images from the PACS and clinical data retrieved from the RIS can be used in conjunction in real-time as well as retrospectively on-demand.
+
+The Niffler framework consists of:
+- On-demand and real-time retrieval and processing of DICOM images from the PACS environment configured to accept requests from a deployment of Niffler.
+- Acquisition and processing of clinical data from a RIS, to enable real-time analytics (RTA).
+- Supportive utility functions such as DICOM → PNG conversion, DICOM → NifTi conversion, DICOM anonymization, and the workflows module.
+- Scanner Usage Visualization with PACS And RIS data algorithm (modules/suvpar).
+- Sample applications of the Niffler modules (modules/app-layer).
+
+Niffler enables receiving DICOM images real-time as a data stream from PACS as well as specific DICOM data based on a series of DICOM C-MOV queries. The Niffler real-time DICOM receiver extracts the metadata free of PHI as the images arrive, store the metadata in a Mongo database, and deletes the images nightly. The on-demand extractor reads a CSV file provided by the user (consisting of a list of values for PatientID, AccessionNumber, or other DICOM keywords), and performs a series of DICOM C-MOVE requests to receive them from the PACS, without manually querying them. Niffler also provides additional features such as converting DICOM images into PNG images, and perform additional computations such as computing scanner utilization and finding scanners with misconfigured clocks.
+
+
+# Configure Niffler
+
+Niffler consists of multiple modules, inside the modules folder. Here we will look into the common configuration and installation steps of Niffler. An introduction to Niffler can be found [here](https://emory-hiti.github.io/Niffler/).
+
+
+## Install Niffler
+
+To deploy Niffler, checkout Niffler source code and run the installation script.
+```
+$ git clone https://github.com/Emory-HITI/Niffler.git
+
+$ cd Niffler
+```
+The master branch is stable whereas the dev branch has the bleeding edge.
+
+You might want to use the dev branch for the latest updates. For more stable version, skip the below step:
+```
+$ git checkout dev
+```
+Finally, run the installation script for a quick installation of dependencies. 
+
+```
+$ sh install.sh
+```
+
+This script is written for Centos. If you are using another operating system, please go through the script and make necessary changes as the script is easy to follow.
+
+Please refer to each module's individual README for additional instructions on configuring, deploying, and using Niffler for each of its modules.
+
+
+## Configure PACS
+
+Both meta-extraction and cold-extraction modules require proper configuration of a PACS environment to allow data transfer and query retrieval to Niffler, respectively.
+
+* If you plan to use meta-extraction and cold-extraction modules, please make sure to configure the PACS to send data to Niffler meta-extraction module's host, port, and AE_Title. 
+
+* Niffler cold-extraction won't receive data unless the PACS allows the requests from Niffler cold-extraction (host/port/AE_Title). Please go through your PACS framework's documentation on configuring host/port/AE_Tiles for a new AE to accept queries from (Query AET). Those instructions should prepare your PACS to receive queries from Niffler as the Query AET.
+
+
+
+# Citing Niffler
+
+If you use Niffler in your research, please cite the below papers:
+
+* Pradeeban Kathiravelu, Puneet Sharma, Ashish Sharma, Imon Banerjee, Hari Trivedi, Saptarshi Purkayastha, Priyanshu Sinha, Alexandre Cadrin-Chenevert, Nabile Safdar, and Judy Wawira Gichoya. **A DICOM Framework for Machine Learning Pipelines against Real-Time Radiology Images.** Journal of Digital Imaging (JDI). August 2021. https://doi.org/10.1007/s10278-021-00491-w
+
+* Pradeeban Kathiravelu, Ashish Sharma, and Puneet Sharma. **Understanding Scanner Utilization With Real-Time DICOM Metadata Extraction.** IEEE Access, 9 (2021): 10621-10633. https://doi.org/10.1109/ACCESS.2021.3050467
+
+* Pradeeban Kathiravelu, Nan Li, Nishchal Singi, Ananth Reddy Bhimireddy, Ryan Birmingham, Judy Wawira Gichoya, Hari Trivedi, Nabile Safdar, Ashish Sharma, and Puneet Sharma. **Visualizing Scanner Utilization from MRI Metadata and Clinical Data.** IEEE Computer. Accepted. 2022 December.

Niffler.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+LICENSE
+README.md
+pyproject.toml
+setup.cfg
+Niffler.egg-info/PKG-INFO
+Niffler.egg-info/SOURCES.txt
+Niffler.egg-info/dependency_links.txt
+Niffler.egg-info/requires.txt
+Niffler.egg-info/top_level.txt
+Niffler.egg-info/zip-safe

Niffler.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

Niffler.egg-info/requires.txt

@@ -0,0 +1,18 @@
+requests
+pymongo
+schedule
+pydicom
+pynetdicom
+image
+numpy
+pandas
+pillow
+pypng
+
+[dev]
+pytest
+pytest-mock
+pytest-cov
+tqdm
+pycryptodomex
+SQLAlchemy

Niffler.egg-info/top_level.txt

@@ -0,0 +1 @@
+

Niffler.egg-info/zip-safe

@@ -0,0 +1 @@
+

README.md

@@ -33,10 +33,16 @@ $ git checkout dev
 ```
 Finally, run the installation script for a quick installation of dependencies. 
 
+For linux use:
 ```
 $ sh install.sh
 ```
 
+For macOS use:
+```
+$ sh macos_install.sh
+```
+
 This script is written for Centos. If you are using another operating system, please go through the script and make necessary changes as the script is easy to follow.
 
 Please refer to each module's individual README for additional instructions on configuring, deploying, and using Niffler for each of its modules.

install.sh

@@ -2,29 +2,64 @@
 echo "Configuring Niffler"
 sudo chmod -R 777 .
 
+name=""
+
+if [[ -e /etc/os-release ]]; then
+    . /etc/os-release
+    name=$NAME
+else
+    name="None"
+fi
+
+echo "Found distribution: $name"
+
 wget -qO- https://get.nextflow.io | bash
 sudo mv nextflow /usr/local/bin
 
 PIP=`head -n 1 init/pip.out`
 if [ "$PIP" = false ] ; then
-    sudo yum install -y python3
-    echo "Installing pip"
-    sudo yum install python3-pip
+    echo "Installing pip and python3..."
+    if [ "$name" = "Fedora" ] ; then
+        sudo yum install -y python3
+        sudo yum install -y python3-pip
+    elif [ "$name" = "Arch Linux" ] ; then
+        sudo pacman -S --noconfirm python
+        sudo pacman -S --noconfirm python-pip
+    elif [ "$name" = "Ubuntu" ] ; then
+        sudo apt-get install -y python3
+        sudo apt-get install -y python3-pip
+    fi
+
     pip install -r requirements.txt
     pip install -i https://test.pypi.org/simple/ HITI-anon-internal
-    wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
-    sh Anaconda3-2020.11-Linux-x86_64.sh -u
-    source ~/.bashrc
-    rm Anaconda3-2020.11-Linux-x86_64.sh
+    if command -v conda >/dev/null 2>&1 ; then
+        echo "Conda is already installed, skipping..."
+    else
+        wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
+        sh Anaconda3-2020.11-Linux-x86_64.sh -u
+        source ~/.bashrc
+        rm Anaconda3-2020.11-Linux-x86_64.sh
+    fi
     echo "true" > init/pip.out
 fi
 
 MISC=`head -n 1 init/misc.out`
 if [ "$MISC" = false ] ; then
     echo "Installing gdcm and mail"
     conda install -c conda-forge -y gdcm
-    sudo yum install mailx -y
-    sudo yum install sendmail sendmail-cf
+
+    if [ "$name" = "Fedora" ] ; then
+        sudo yum install mailx -y
+        sudo yum install sendmail sendmail-cf
+    elif [ "$name" = "Arch Linux" ] ; then
+        sudo pacman -S --noconfirm mailx
+        sudo pacman -S --noconfirm sendmail sendmail-cf
+
+    elif [ "$name" = "Ubuntu" ] ; then
+        sudo apt-get install -y mailutils
+        sudo apt-get install -y sendmail sendmail-cf
+    fi
+
     chmod +x modules/meta-extraction/service/mdextractor.sh
     echo "Disable THP"
     sudo cp init/disable-thp.service /etc/systemd/system/disable-thp.service
@@ -36,11 +71,19 @@ fi
 
 DCM4CHE=`head -n 1 init/dcm4che.out`
 if [ "$DCM4CHE" = false ] ; then
-    echo "Installing JDK"
-    sudo yum install java-1.8.0-openjdk-devel
-    echo "Installing Maven"
-    sudo dnf install maven
-    echo "Installing DCM4CHE"
+    echo "Installing jdk and Maven..."
+    if [ "$name" = "Fedora" ] ; then
+        sudo yum install java-1.8.0-openjdk-devel
+        sudo dnf install maven
+    elif [ "$name" = "Arch Linux" ] ; then
+        sudo pacman -S --noconfirm jdk8-openjdk
+        sudo pacman -S --noconfirm maven
+
+    elif [ "$name" = "Ubuntu" ] ; then
+        sudo apt install -y openjdk-8-jdk
+        sudo apt install -y maven
+    fi
+
     cd ..
     wget https://sourceforge.net/projects/dcm4che/files/dcm4che3/5.22.5/dcm4che-5.22.5-bin.zip/download -O dcm4che-5.22.5-bin.zip
     unzip dcm4che-5.22.5-bin.zip
@@ -51,9 +94,18 @@ fi
 
 MONGO=`head -n 1 init/mongo.out`
 if [ "$MONGO" = false ] ; then
-    echo "Installing mongo"
+    echo "Installing MongoDb..."
     sudo cp init/mongodb-org-4.2.repo /etc/yum.repos.d/
-    sudo yum install mongodb-org
+
+    if [ "$name" = "Fedora" ] ; then
+        sudo yum install mongodb-org
+    elif [ "$name" = "Arch Linux" ] ; then
+        sudo pacman -S --noconfirm mongodb
+
+    elif [ "$name" = "Ubuntu" ] ; then
+        sudo apt install -y mongodb
+    fi
+
     sudo systemctl start mongod
     sudo systemctl enable mongod
     mongo init/mongoinit.js
@@ -65,6 +117,7 @@ if [ "$MONGO" = false ] ; then
 fi
 
 SERVICE=`head -n 1 init/service.out`
+
 if [ "$SERVICE" = false ] ; then
     echo "Installing Niffler Frontend"
     pip install -r modules/frontend/requirements.txt

macos_install.sh

@@ -0,0 +1,81 @@
+#!/bin/sh
+
+echo "Configuring Niffler"
+sudo chmod -R 777 .
+
+curl -s https://get.nextflow.io | bash
+sudo mv nextflow /usr/local/bin
+
+PIP=`head -n 1 init/pip.out`
+if [ "$PIP" = false ] ; then
+    brew install python3
+    echo "Installing pip"
+    python3 -m ensurepip
+    pip3 install -r requirements.txt
+    pip3 install -i https://test.pypi.org/simple/ HITI-anon-internal
+    wget https://repo.anaconda.com/archive/Anaconda3-2022.10-MacOSX-arm64.sh
+    sh Anaconda3-2022.10-MacOSX-arm64.sh -u
+    source ~/.zshrc
+    rm Anaconda3-2022.10-MacOSX-arm64.sh
+    echo "true" > init/pip.out
+fi
+
+MISC=`head -n 1 init/misc.out`
+if [ "$MISC" = false ] ; then
+    echo "Installing gdcm and mail"
+    brew install gdcm
+    # mailx not availlable for MacOs
+    brew install mailutils
+    # sendmail-cf not available for MacOS
+    chmod +x modules/meta-extraction/service/mdextractor.sh
+    # echo "Disable THP"
+    # sudo cp init/disable-thp.service /etc/systemd/system/disable-thp.service
+    # sudo systemctl daemon-reload
+    # sudo systemctl start disable-thp
+    # sudo systemctl enable disable-thp
+    echo "true" > init/misc.out
+fi
+
+
+DCM4CHE=`head -n 1 init/dcm4che.out`
+if [ "$DCM4CHE" = false ] ; then
+    echo "Installing JDK"
+    brew tap adoptopenjdk/openjdk
+    brew install --cask adoptopenjdk8
+    echo "Installing Maven"
+    brew install maven
+    echo "Installing DCM4CHE"
+    cd ..
+    wget https://sourceforge.net/projects/dcm4che/files/dcm4che3/5.22.5/dcm4che-5.22.5-bin.zip/download -O dcm4che-5.22.5-bin.zip
+    unzip dcm4che-5.22.5-bin.zip
+    rm dcm4che-5.22.5-bin.zip
+    cd Niffler
+    echo "true" > init/dcm4che.out
+fi
+
+MONGO=`head -n 1 init/mongo.out`
+if [ "$MONGO" = false ] ; then
+    echo "Installing mongo"
+    brew tap mongodb/brew
+    brew install mongodb-community
+    brew services start mongodb-community
+    brew services enable mongodb-community
+    mongosh init/mongoinit.js
+    sudo cp modules/meta-extraction/service/mdextractor.plist /Library/LaunchDaemons/
+    sudo launchctl load /Library/LaunchDaemons/mdextractor.plist
+    sudo launchctl enable system/mdextractor.plist
+    echo "true" > init/mongo.out
+fi
+
+SERVICE=`head -n 1 init/service.out`
+if [ "$SERVICE" = false ] ; then
+    echo "Installing Niffler Frontend"
+    pip install -r modules/frontend/requirements.txt
+    pip install -i https://test.pypi.org/simple/ HITI-anon-internal
+    chmod +x modules/frontend/service/frontend_service.sh
+    sudo cp modules/frontend/service/niffler.plist /Library/LaunchDaemons/
+    sudo launchctl load -w /Library/LaunchDaemons/niffler.plist
+    sudo launchctl enable system/niffler.plist
+    echo "true" > init/service.out
+fi
+