diff --git a/README.md b/README.md index eb13a62c..726be0cd 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ # Gregory + Gregory aggregates searches in JSON and outputs to a Hugo static site # Sources for searches @@ -16,88 +17,90 @@ Gregory aggregates searches in JSON and outputs to a Hugo static site # Live Version -https://gregory-ms.com + # Install ## Server Requirements -- [ ] [Docker](https://www.docker.com/) and [docker-compose](https://docs.docker.com/compose/) with 2GB of swap memory to be able to build the MachineLearning Models. [Adding swap for Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04) -- [ ] [Hugo](https://gohugo.io/) +- [ ] [Docker](https://www.docker.com/) and [docker-compose](https://docs.docker.com/compose/) with 2GB of swap memory to be able to build the MachineLearning Models. [Adding swap for Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04) +- [ ] [Hugo](https://gohugo.io/) - [ ] [Mailgun](https://www.mailgun.com/) ## Setup the environment -Run `pip3 install -r python-ml/requirements.txt` - -Execute `python3 setup.py`. The script will check if you have all the requirements and run the Node-RED container. +1. Edit the .env file to reflect your settins and credentials. +2. Execute `python3 setup.py`. The script will check if you have all the requirements and run the Node-RED container. +3. Run `sudo docker-compose up -d` to fetch the required images and run the -Edit the .env file's variables to reflect your environment. +The final website is built by running `python3 ./build.py`. -Finally, build the site with `python3 ./build.py`. +## Running django -# Node-RED +Once the db container is running, start the admin container and run the following: -This is where all of the processing of information happens. Node-RED provides a set of API endpoints to access the information it collects. +```bash +sudo docker exec -it admin /bin/bash +python manage.py makemigrations +python manage.py migrate +python manage.py createsuperuser +``` -The following tabs have been configured to divide the flows: +Now you can login at or wherever your reverse proxy is listening on. -1. Research (Collect data from different sources) -2. Email Digest (Sent to the admin and to subscribers) -3. API -4. Tests +# How everything fits together -`data/articles.json` and `data/trials.json` are generated from a Node-Red flow available in the `flows.json` file. +## Node-RED -Node-RED is installed with a custom dockerfile from their official repository. https://github.com/node-red/node-red-docker/tree/master/docker-custom +We are using [Node-RED](https://nodered.org/) to collect articles and clinical trials. -# Mailgun +Node-RED is installed with a custom dockerfile that includes some Machine Learning and Artificial Intelligence python modules, in case we need them. It is derived from the [Node-Red repository](https://github.com/node-red/node-red-docker/tree/master/docker-custom). -Currently, we are using Mailgun to send emails to the admin and subscribers. These nodes can be found on the Email Digest tab of Node-RED and have been disabled. +## Django and Postgres -To enable them, you will need a mailgun account, or you can replace them with another way to send emails. +The database and the API are managed using the `admin` container that comes with [Django](https://www.djangoproject.com/) and the [Django Rest Framework](https://www.django-rest-framework.org/) installed. -# Database +The database is running on a postgres container called `db`. -The path https://api.gregory-ms.com/articles/all and https://api.gregory-ms.com/trials/all includes the full database export. +## Metabase -The same information is available in excel and json format: https://gregory-ms.com/downloads/ +A database is only as good as it's ability to answer questions. We have a separate `metabase` container in the docker-compose.yaml file that connects directly to Gregory's database. -# Update the Machine Learning Algorithms +It's available at -1. `cd docker-python; source .venv/bin/activate` -2. `python3 1_data_processor.py` -3. `python3 2_train_models.py` -4. Login to sqlite3: `sqlite3 gregory/docker-data/gregory.db` -5. Reset the Machine Learning records with `UPDATE articles SET ml_prediction_gnb = NULL, ml_prediction_lr = NULL WHERE article_id > 0;` -6. The Node-Red flow to review the articles runs every 10 minutes. +The current website is also using some embeded dashboards whose keys are produced each time you run `build.py`. An example can be found in the [MS Observatory Page](https://gregory-ms.com/observatory/) +## Mailgun +Email are sent from the `admin` container using Mailgun. +To enable them, you will need a mailgun account, or you can replace them with another way to send emails. -## Running django - -Once the db container is running, start the admin container and run the following: +You need to configure the relevant variables for this to work: ```bash -sudo docker exec -it admin /bin/bash -python manage.py makemigrations -python manage.py migrate -python manage.py createsuperuser +EMAIL_USE_TLS=true +EMAIL_MAILGUN_API='YOUR API KEY' +EMAIL_DOMAIN='YOURDOMAIN' +EMAIL_MAILGUN_API_URL="https://api.eu.mailgun.net/v3/YOURDOMAIN/messages" ``` -Now you can login at https://YOUR-SUB.DOMAIN/admin or wherever your reverse proxy is listening on. +# Update the Machine Learning Algorithms +This is not working right now and there is a [pull request to setup an automatic process to keep the machine learning models up to date](https://github.com/brunoamaral/gregory/pull/110). + +1. `cd docker-python; source .venv/bin/activate` +2. `python3 1_data_processor.py` +3. `python3 2_train_models.py` # Thank you to @[Antoniolopes](https://github.com/antoniolopes) for helping with the Machine Learning script. -@[Chbm](https://github.com/chbm) for help in keeping the code secure. -@[Malduarte](https://github.com/malduarte) for help with the migration from sqlite to postgres. -@[Jneves](https://github.com/jneves) for help with the build script -@[Melo](https://github.com/melo) for showing me [Hugo](https://github.com/gohugoio/hugo) -@[Nurv](https://github.com/nurv) for the suggestion in using Spacy.io -@[Rcarmo](https://github.com/rcarmo) for showing me [Node-RED](https://github.com/node-red/node-red) +@[Chbm](https://github.com/chbm) for help in keeping the code secure. +@[Malduarte](https://github.com/malduarte) for help with the migration from sqlite to postgres. +@[Jneves](https://github.com/jneves) for help with the build script +@[Melo](https://github.com/melo) for showing me [Hugo](https://github.com/gohugoio/hugo) +@[Nurv](https://github.com/nurv) for the suggestion in using Spacy.io +@[Rcarmo](https://github.com/rcarmo) for showing me [Node-RED](https://github.com/node-red/node-red) And the **Lobsters** at [One Over Zero](https://github.com/oneoverzero) -